systems_performance_observability

¹⁾

²⁾

unsigned int) ((REC→dev) & ((1U « 20) - 1

³⁾

nsecs - @ts tid) / 1000000); delete(@ts tid); } END { clear(@ts); }' Attaching 3 probes…

@sleep_ms: [0] 1280 ]] | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [1] 1 | systems_performance_observability_tools | [2, 4) 1 | systems_performance_observability_tools | [4, 8) 0 | systems_performance_observability_tools | [8, 16) 0 | systems_performance_observability_tools | [16, 32) 0 | systems_performance_observability_tools | [32, 64) 0 | systems_performance_observability_tools | [64, 128) 0 | systems_performance_observability_tools | [128, 256) 0 | systems_performance_observability_tools | [256, 512) 0 | systems_performance_observability_tools | [512, [[1K) 2 ]] | systems_performance_observability_tools | The [[output shows that do_nanosleep() was usually a fast function, returning in zero milliseconds (round ed down) 1,280 times. Two occurrences reached the 512 to 1024 millisecond range. bpftrace syntax is explained in Chapter 15, BPF, which includes a similar example for timing vfs_read(). kprobes Interface and Overhead The kprobes interface is similar to tracepoints. There is a way to instrument them via /sys files, via the perf_event_open(2) syscall (which is preferred), and also via the register_kprobe() kernel API. The overhead is similar to that of tracepoints when the entries to functions are traced (Ftrace method, if available), and higher when function offsets are traced (break point method) or when kretprobes are used (trampoline method). For a particular system I measured the minimum kprobe CPU cost to be 76 nano seconds, and the minimum kretprobe CPU cost to be 212 nano seconds Gregg 19]. kprobes Documentation kprobes are documented in the Linux source under Documentation/kprobes.txt. The kernel functions they instrument are typically not documented outside of the kernel source (since most are not an API, they don’t need to be). I summarized advanced kprobe topics in BPF Performance Tools, Chapter 2 Gregg 19]: how they work at the instruction level. 4.3.7 uprobes uprobes (user-space probes) are similar to kprobes, but for user-space. They can dynamically instrument functions in applications and libraries, and provide an unstable API for diving deep into software internals beyond the scoped of other tools. uprobes were made available in Linux 3.5, released in 2012. uprobes can be used by the tracers introduced in Section 4.5, Tracing Tools, and covered in depth in Chapters 13 to 15. uprobes Example As an example of uprobes, the following bpftrace command lists possible uprobe function entry locations in the bash(1) shell: Click here to view code image

bpftrace -l 'uprobe:/bin/bash:*'

uprobe:/bin/bash:rl_old_menu_complete uprobe:/bin/bash:maybe_make_export_env uprobe:/bin/bash:initialize_shell_builtins uprobe:/bin/bash:extglob_pattern_p uprobe:/bin/bash:dispose_cond_node uprobe:/bin/bash:decode_prompt_string [..] The full output show ed 1,507 possible uprobes. uprobes instrument code and create uprobe events when needed (a uprobe-based trace event): the user-space code runs unmodified by default. This is similar to using a debugger to add a break point to a function: before the break point is added, the function is running unmodified. uprobes Arguments Arguments to user functions are made available by uprobes. As an example, the following uses bpftrace to instrument the decode_prompt_string() bash function and print the first argument as a string: Click here to view code image

bpftrace -e 'uprobe:/bin/bash:decode_prompt_string { printf(”%s\n“, str(arg0)); }'

Attaching 1 probe… \[\e[31;1m\]\u@\h:\w>\[\e[0m\] \[\e[31;1m\]\u@\h:\w>\[\e[0m\]

The output shows the bash(1) prompt string on this system. The uprobe for decode_prompt_string() is created when the bpftrace program begins running, and is removed when bpftrace terminates (Ctrl-C). uretprobes The return from user functions and their return value can be traced using uretprobes (short for user-level return probes), which are similar to uprobes. When paired with uprobes and a tracer that records timestamps, the duration of a user-level function can be measured. Be aware that the overhead of uretprobes can significantly skew such measurements of fast functions. uprobes Interface and Overhead The uprobes interface is similar to kprobes. There is a way to instrument them via /sys files and also (preferably) via the perf_event_open(2) syscall. uprobes currently work by trapping into the kernel. This costs much higher CPU overheads than kprobes or tracepoints. For a particular system I measured, the minimum uprobe cost 1,287 nano seconds, and the minimum uretprobe cost 1,931 nano seconds Gregg 19]. The uretprobe overhead is higher because it is a uprobe plus a trampoline function. uprobe Documentation uprobes are documented in the Linux source under Documentation/trace/uprobe tracer.rst. I summarized advanced uprobe topics in BPF Performance Tools, Chapter 2 Gregg 19]: how they work at the instruction level. The user functions they instrument are typically not documented outside of the application source (since most are unlikely to be an API, they don’t need to be). For documented user-space tracing, use USDT. 4.3.8 USDT User-level statically-defined tracing (USDT) is the user-space version of tracepoints. USDT is to uprobes as tracepoints is to kprobes. Some applications and libraries have added USDT probes to their code, providing a stable (and documented) API for tracing application-level events. For example, there are USDT probes in the Java JDK, in the PostgreSQL database, and in libc. The following lists OpenJDK USDT probes using bpftrace: Click here to view code image

bpftrace -lv 'usdt:/usr/lib/jvm/openjdk/lib jvm.so:*'

usdt:/usr/lib/jvm/openjdk/lib jvm.so:hotspot:class__load ed usdt:/usr/lib/jvm/openjdk/lib jvm.so:hotspot:class__unload ed usdt:/usr/lib/jvm/openjdk/lib jvm.so:hotspot:method__compile__begin usdt:/usr/lib/jvm/openjdk/lib jvm.so:hotspot:method__compile__end usdt:/usr/lib/jvm/openjdk/lib jvm.so:hotspot:gc__begin usdt:/usr/lib/jvm/openjdk/lib jvm.so:hotspot:gc__end […] This lists USDT probes for Java [[class loading and unloading, method compilation, and garbage collection. Many more were truncated: the full listing shows 524 USDT probes for this JDK version. Many applications already have custom event logs that can be enabled and configured, and are useful for performance analysis. What makes USDT probes different is that they can be used from various tracers that can combine application context with kernel events such as disk and network I/O. An application-level logger may tell you that a database query was slow due to file [[system I/O, but a tracer can reveal more information: e.g., the query was slow due to lock contention in the file system, and not disk I/O as you might have assumed. Some applications contain USDT probes, but they are not currently enabled in the packaged version of the application (this is the case with OpenJDK). Using them requires re building the application from source with the appropriate config option. That option may be called –enable-dtrace-probes after the D[[Trace tracer, which drove ad option of USDT in applications. USDT probes must be compiled into the executable they instrument. This is not possible for JIT-compiled languages such as Java, which usually compiles on the fly. A solution to this is dynamic USDT, which precompiles probes as a shared library, and provides an interface to call them from the JIT-compiled language. Dynamic USDT libraries exist for Java, Node.js, and other languages. Interpreted languages have a similar problem and need for dynamic USDT. USDT probes are implemented in Linux using uprobes: see the previous section for a description of uprobes and their overhead. In addition to the enabled overhead, USDT probes place nop instructions in the code, as do tracepoints. USDT probes can be used by the tracers introduced in Section 4.5, Tracing Tools, covered in depth in Chapters 13 to 15 (although using USDT with Ftrace requires some extra work). USDT Documentation If an application makes USDT probes available, they should be documented in the application’s documentation. I summarized advanced USDT topics in BPF Performance Tools, Chapter 2 Gregg 19]: how USDT probes can be added to application code, how they work internally, and dynamic USDT. 4.3.9 Hardware Counters (PMCs) The processor and other devices commonly support hardware counters for observing activity. The main source are the processors, where they are commonly called performance monitoring counters (PMCs). They are known by other names as well: CPU performance counters (CPCs), performance instrumentation counters (PICs), and performance monitoring unit events (PMU events). These all refer to the same thing: programmable hardware registers on the processor that provide low-level performance information at the CPU cycle level. PMCs are a vital resource for performance analysis. Only through PMCs can you measure the efficiency of CPU instructions, the hit ratios of CPU caches, the utilization of memory and device buses, interconnect utilization, stall cycles, and so on. Using these to analyze performance can lead to various performance optimizations. PMC Examples While there are many PMCs, Intel have selected seven as an “architectural set,” which provide a high-level overview of some core functions Intel 16]. The presence of these architectural set PMCs can be checked using the cpu id instruction. Table 4.4 shows this set, which serves as an example set of useful PMCs. Table 4.4 Intel architectural PMCs As an example of PMCs, if you run the perf stat command without specifying events (no -e), it defaults to instrumenting]] the architectural PMCs. For example, the following runs perf stat on the gzip(1) command: Click here to view code image

perf stat gzip words

Performance counter stats for 'gzip words': 156.927428 task-clock (msec) # 0.987 CPUs utilized 1 context-switches # 0.006 K/sec 0 cpu-migrations # 0.000 K/sec 131 page-faults # 0.835 K/sec 209,911,358 cycles # 1.338 GHz 288,321,441 instructions # 1.37 insn per cycle 66,240,624 branches # 422.110 M/sec 1,382,627 branch-misses # 2.09% of all branches 0.159065542 seconds time elapsed The raw counts are the first column; after a hash are some statistics, including an important performance metric: instructions per cycle (insn per cycle). This shows how efficiently the CPUs are executing instructions — the higher, the better. This metric is explained in the Chapter 6, CPUs, Section 6.3.7, IPC, CPI. PMC Interface On Linux PMCs are accessed via the perf_event_open(2) syscall and are consumed by tools including perf(1). While there are hundreds of PMCs available, there is only a fixed number of registers available in the CPUs to measure them at the same time, perhaps as few as six. You need to choose which PMCs you’d like to measure on those six registers, or cycle through different PMC sets as a way of sampling them (Linux perf(1) supports this automatically). Other software counters do not suffer from these constraints. PMCs can be used in different modes: counting, where they count events with practically zero overhead; and overflow sampling, where an interrupt is raised for one in every configurable number of events, so that state can be captured. Counting can be used to quantify problems; overflow sampling can be used to show the code path responsible. perf(1) can perform counting using the stat subcommand, and sampling using the record subcommand; see Chapter 13, perf. PMC Challenges Two common challenges when using PMCs are their accuracy for overflow sampling and their availability in cloud environments. Overflow sampling may not record the correct instruction pointer that triggered the event, due to interrupt latency (often called “skid”) or out-of-order instruction execution. For CPU cycle profiling, such skid may not be a problem, and some profilers deliberately introduce jitter to avoid lock step sampling (or use an offset sampling rate such as 99 Hertz). But for measuring other events, such as LLC misses, the sampled instruction pointer needs to be ac curate. The solution is processor support for what are known as precise events. On Intel, precise events use a technology called precise event-based sampling (PEBS),9 which uses hardware buffers to record a more ac curate (“precise”) instruction pointer at the time of the PMC event. On AMD, precise events use instruction-based sampling (IBS) [Drongowski 07]. The Linux perf(1) command supports precise events (see Chapter 13, perf, Section 13.9.2, CPU Profiling). 9Some of Intel’s documentation expands PEBS differently, as: processor event-based sampling. Another challenge is cloud computing, as many cloud environments disable PMC access for their guests. It is technically possible to enable it: for example, the Xen hypervisor has the vpmu command line option, which allows different sets of PMCs to be exposed to guests10 [Xenbits 20]. Amazon have enabled many PMCs for their Nitro hypervisor guests.11 Also, some cloud providers offer “bare-metal instances” where the guest has full processor access, and therefore full PMC access. 10I wrote the Xen code that allows different PMC modes: “ipc” for instructions-per-cycle PMCs only, and “arch” for the Intel architectural set. My code was just a firewall on the existing vpmu support in Xen. 11Currently only for larger Nitro instances where the VM owns a full processor socket (or more). PMCs Documentation PMCs are processor-specific and documented in the appropriate processor software developer’s manual. Examples by processor manufacturer]]:

	[[Intel]]: [[Chapter]] 19, “[[Performance Monitoring]] [[Event]]s,” of [[Intel]]® 64 and [[IA]]-32 [[Architecture]]s [[Software]] [[Developer]]’s [[Manual]] [[Volume]] 3 [[Intel]] 16].

	[[AMD]]: [[Section]] 2.1.1, “[[Performance Monitor]] [[Counter]]s,” of [[Open-Source]] [[Register]] [[Reference]] For [[AMD]] [[Family]] 17h [[Processor]]s [[Model]]s 00h-2Fh [[AMD]] 18]

	[[ARM]]: [[Section]] D7.10, “PMU [[Event]]s and [[Event]] [[Number]]s,” of [[Arm]]® [[Architecture]] [[Reference Manual]] Armv8, for Armv8-A [[architecture]] [[profile]] [[ARM]] 19]

There has been work to develop a standard naming scheme for PMCs that could be supported across all processors, called the performance application programming interface (PAPI) [UTK 20]. Operating system support for PAPI has been mixed: it requires frequent updates to map PAPI names to vendor PMC codes. Chapter 6, CPUs, Section 6.4.1, Hardware, sub section Hardware Counters (PMCs), describes their implementation in more detail and provides additional PMC examples. 4.3.10 Other Observability Sources Other observability sources include:

	MSRs: [[PMC]]s are [[implemented]] [[using]] [[model]]-specific [[register]]s (MSRs). There are other MSRs for [[show]]ing the [[configuration]] and [[health]] of the [[system]], including the [[CPU]] [[clock rate]], [[usage]], [[temp]]eratures, and [[power]] [[consumption]]. The [[available]] MSRs are [[dependent]] on the [[processor]] [[type]] ([[model]]-specific), [[BIOS]] [[version]] and [[setting]]s, and [[hypervisor]] [[setting]]s. One [[use]] is an [[ac]][[curate]] [[cycle]]-[[based]] [[measurement]] of [[CPU]] utilization]].

	p[[trace]](2): This [[syscall]] [[control]]s [[process]] [[tracing]], which is used by [[gdb]](1) for [[process]] [[debugging]] and [[strace]](1) for [[tracing]] [[syscall]]s. It is [[break]][[point]]-[[based]] and can [[slow]] the [[target]] over one hundred-fold. [[Linux]] also has [[tracepoint]]s, [[introduce]]d in [[Section]] 4.3.5, [[Tracepoint]]s, for more [[efficient]] [[syscall]] [[tracing]].

	[[Function]] [[profiling]]: [[Profiling]] [[function call]]s (m[[count]]() or __f[[entry]]__()) are [[added]] to the [[start]] of all non-[[inlined]] [[kernel]] [[function]]s on [[x86]] for [[efficient]] [[Ftrace]] [[function]] [[tracing]]. They are [[convert]][[ed]] to nop [[instruction]]s until [[needed]]. See [[Chapter]] 14, [[Ftrace]].

	[[Network]] [[sniffing]] (libpcap): These [[interface]]s [[provide]] a way to [[capture]] [[packet]]s from [[network]] [[device]]s for [[detailed]] [[investigation]]s into [[packet]] and [[protocol]] [[performance]]. On [[Linux]], [[sniffing]] is [[provided]] via the libpcap [[library]] and /[[proc]]/[[net]]/[[dev]] and is [[consumed]] by the [[tcpdump]](8) [[tool]]. There are [[overhead]]s, both [[CPU]] and [[storage]], for [[capturing]] and [[examining]] all [[packet]]s. See [[Chapter]] 10 for more about [[network]] [[sniffing]].

	[[net[[filter]] [[conn[[track]]: The [[Linux]] [[net[[filter]] [[technology]] [[allow]]s [[custom]] [[handler]]s to be [[executed]] on [[event]]s, not just for [[firewall]], but also for [[connection]] [[track]]ing]] ([[conn[[track]]). This [[allow]]s [[log]]s to be [[created]] of [[network]] [[flow]]s [Ayuso 12].

	[[Process]] [[accounting]]: This [[date]]s [[back]] to [[mainframe]]s and the [[need]] to [[bill]] [[department]]s and [[user]]s for their [[computer]] [[usage]], [[based]] on the [[execution]] and [[runtime]] of [[process]]es. It [[exist]]s in some [[form]] for [[Linux]] and other [[system]]s and can some[[time]]s be [[helpful]] for [[performance analysis]] at the [[process]] [[level]]. For [[example]], the [[Linux]] [[atop]](1) [[tool]] [[use]]s [[process]] [[accounting]] to [[catch]] and [[display]] [[information]] from [[short]]-[[live]]d [[process]]es that would otherwise be missed when taking [[snapshot]]s of /[[proc]] [Atop[[tool]] 20].

	[[Software]] [[event]]s: These are [[related]] to [[hardware]] [[event]]s but are [[instrument]][[ed]] in [[software]]. [[Page fault]]s are an [[example]]. [[Software]] [[event]]s are made [[available]] via the [[perf]]_[[event]]_[[open]](2) [[interface]] and are used by [[perf]](1) and [[bpftrace]]. They are [[picture]]d in [[Figure]] 4.5.

	[[System call]]s: Some [[system]] or [[library]] [[call]]s may be [[available]] to [[provide]] some [[performance metric]]s. These [[include]] getr[[usage]](2), a [[system call]] for [[process]]es to [[get]] their own [[resource]] [[usage]] [[statistic]]s, including [[user]]- and [[system]]-[[time]], [[fault]]s, [[message]]s, and [[context]] [[switch]]es.

If you are interest ed in how each of these works, you will find that documentation is usually available, intended for the developer who is building tools upon these interfaces. And More Depending on your kernel version and enabled options, even more observability sources may be available. Some are mentioned in later chapters of this book. For Linux these include I/O accounting, blk trace, timer_stats, lock stat, and debug fs. One way to find such sources is to read the kernel code you are interest ed in observing and see what statistics or tracepoints have been placed there. In some cases there may be no kernel statistics for what you are after. Beyond dynamic instrumentation (Linux kprobes and uprobes), you may find that debuggers such as gdb(1) and lldb(1) can fetch kernel and application variables to shed some light on an investigation. Solaris Kstat As an example of a different way to provide system statistics, Solaris-based systems use a kernel statistics (Kstat) framework that provides a consistent hierarchical structure of kernel statistics, each named using the following four-tuple: Click here to view code image module:instance:name:statistic These are

	[[module]]: This usually refers to the [[kernel]] [[module]] that [[created]] the [[statistic]], such as sd for the [[SCSI]] [[disk]] [[driver]], or [[zfs]] for the [[ZFS file system]].

	[[instance]]: Some [[module]]s [[exist]] as [[multiple]] [[instance]]s, such as an sd [[module]] [[for each]] [[SCSI]] [[disk]]. The [[instance]] is an [[enumeration]].

	[[name]]: This is a [[name]] for the [[group]] of [[statistic]]s.

	[[statistic]]: This is the [[individual]] [[statistic]] [[name]].

Kstats are accessed using a binary kernel interface, and various libraries exist. As an example Kstat, the following reads the “nproc” statistic using kstat(1M) and specifying the full four-tuple: Click here to view code image $ kstat -p unix:0:system_misc:nproc unix:0:system_misc:nproc 94 This statistic shows the currently running number of processes. In comparison, the /proc/stat-style sources on Linux have in[[consistent formatting and usually require text parsing to process, costing some CPU cycles. 4.4 sar sar(1) was introduced in Section 4.2.4, Monitoring, as a key monitoring facility. While there has been much excitement recently with BPF tracing super powers (and I’m partly responsible), you should not overlook the utility of sar(1) — it’s an essential systems performance tool that can solve many performance issues on its own. The Linux version of sar(1) is also well-designed, having self-descriptive column headings, network metric groups, and detailed documentation (man pages). sar(1) is provided via the sysstat package. 4.4.1 sar(1) Coverage Figure 4.6 shows the observability coverage from the different sar(1) command line options. Figure 4.6 Linux sar(1) observability This figure shows that sar(1) provides broad coverage of the kernel and devices, and even has observability for fans. The -m (power management) option also supports other arguments not shown in this figure, including IN for voltage inputs, TEMP for device temperatures, and USB for USB device power statistics. 4.4.2 sar(1) Monitoring You may find that sar(1) data collecting (monitoring) is already enabled for your Linux systems. If it isn’t, you need to enable it. To check, simply run sar without options. For example: Click here to view code image $ sar Cannot open /var/log/sysstat/sa19: No such file or directory Please check if data collecting is enabled The output shows that sar(1) data collecting is not yet enabled on this system (the sa19 file refers to the daily archive for the 19th of the month). The steps to enable it may vary based on your distribution. Configuration (Ubuntu) On this Ubuntu system, I can enable sar(1) data collecting by editing the /etc/default/sysstat file and setting ENABLED to be true: Click here to view code image ubuntu

vi /etc/default/sysstat

Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat

and /etc/cron.daily/sysstat files

Should sadc collect system activity informations? Valid values

are ”true“ and ”false“. Please do not put other values, they

will be overwritten by deb conf!

ENABLED=”true“ And then re starting sysstat using: Click here to view code image ubuntu

service sysstat restart

The schedule of statistic recording can be modified in the crontab file for sysstat: Click here to view code image ubuntu

cat /etc/cron.d/sysstat

The first element of the path is a directory where the debian-sa1

script is located

PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

Activity reports every 10 minutes everyday

5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

Additional run at 23:59 to rotate the statistics file

59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2 The syntax 5-55/10 means it will record every 10 minutes for the minute range 5 to 55 minutes past the hour. You can adjust this to suit the resolution desired: the syntax is documented in the crontab(5) man page. More frequent data collection will increase the size of the sar(1) archive files, which can be found in /var/log/sysstat. I often change data collection to: Click here to view code image

/5 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1 -S ALL

The */5 will record every five minutes, and the -S ALL will record all statistics. By default sar(1) will record most (but not all) statistic groups. The -S ALL option is used to record all statistic groups — it is passed to sadc(1), and documented in the man page for sadc(1). There is also an extended version, -S XALL, which records additional break downs of statistics. Reporting sar(1) can be executed with any of the options shown in Figure 4.6 to report the selected statistic group. Multiple options can be specified. For example, the following reports CPU statistics (-u), TCP (-n TCP), and TCP errors (-n ETCP): Click here to view code image $ sar -u -n TCP,ETCP Linux 4.15.0-66-generic (bgregg) 01/19/2020 _x86_64]]_ (8 CPU) 10:40:01 AM CPU %user %nice %system %iowait %steal %idle 10:45:01 AM all 6.87 0.00 2.84 0.18 0.00 90.12 10:50:01 AM all 6.87 0.00 2.49 0.06 0.00 90.58 […] 10:40:01 AM active/s passive/s iseg/s oseg/s 10:45:01 AM 0.16 0.00 10.98 9.27 10:50:01 AM 0.20 0.00 10.40 8.93 […] 10:40:01 AM atmptf/s estres/s retrans/s isegerr/s orsts/s 10:45:01 AM 0.04 0.02 0.46 0.00 0.03 10:50:01 AM 0.03 0.02 0.53 0.00 0.03 […] The first line of output is a system summary, showing the kernel type and version, the hostname, the date, processor architecture, and number of CPUs. Running sar -A will dump all statistics. Output Formats The sysstat package comes with an sadf(1) command for viewing sar(1) statistics in different formats, including JSON, SVG, and CSV. The following examples emit the TCP (-n TCP) statistics in these formats. JSON (-j): JavaScript Object Notation (JSON) can be easily parsed and import ed by many programming languages, making it a suitable output format when building other software upon sar(1). Click here to view code image $ sadf -j – -n TCP {”sysstat“: { ”hosts“: [ { ”node name“: “bgregg”, ”sys name“: ”Linux“, ”release“: “4.15.0-66-generic”, ”machine“: ”x86_64]]“, ”number-of-cpus“: 8, ”file-date“: ”2020-01-19“, ”file-utc-time“: “18:40:01”, ”statistics“: [ { ”timestamp“: {”date“: ”2020-01-19“, ”time“: “18:45:01”, ”utc“: 1, ”interval“: 300}, ”network“: { ”net-tcp“: {”active“: 0.16, ”passive“: 0.00, “iseg”: 10.98, “oseg”: 9.27} } }, […] You can process the JSON output at the command line using the jq(1) tool. SVG (-g): sadf(1) can emit Scalable Vector Graphics (SVG) files that can be viewed in web browsers. Figure 4.7 shows an example. You can use this output format to build rudimentary dashboards. Figure 4.7 sar(1) sadf(1) SVG output12 12Note that I edited the SVG file to make this figure more legible, changing colors and increasing font sizes. CSV (-d): The comma-separated values (CSV) format is intended for import by databases (and uses a semicolon): Click here to view code image $ sadf -d – -n TCP

hostname;interval;timestamp;active/s;passive/s;iseg/s;oseg/s

bgregg;300;2020-01-19 18:45:01 UTC;0.16;0.00;10.98;9.27 bgregg;299;2020-01-19 18:50:01 UTC;0.20;0.00;10.40;8.93 bgregg;300;2020-01-19 18:55:01 UTC;0.12;0.00;9.27;8.07 […] 4.4.3 sar(1) Live When executed with an interval and optional count, sar(1) does live reporting. This mode can be used even when data collection is not enabled. For example, showing the TCP statistics with an interval of one second and a count of five: Click here to view code image $ sar -n TCP 1 5 Linux 4.15.0-66-generic (bgregg) 01/19/2020 _x86_64]]_ (8 CPU) 03:09:04 PM active/s passive/s iseg/s oseg/s 03:09:05 PM 1.00 0.00 33.00 42.00 03:09:06 PM 0.00 0.00 109.00 86.00 03:09:07 PM 0.00 0.00 107.00 67.00 03:09:08 PM 0.00 0.00 104.00 119.00 03:09:09 PM 0.00 0.00 70.00 70.00 Average: 0.20 0.00 84.60 76.80 Data collecting is intended for long intervals, such as five or ten minutes, whereas live reporting allows you to look at per-second variation. Later chapters include various examples of live sar(1) statistics. 4.4.4 sar(1) Documentation The sar(1) man page documents the individual statistics and includes SNMP names in square brackets. For example: Click here to view code image $ man sar […] active/s The number of times TCP connections have made a direct transition to the SYN-SENT state from the CLOSED state per second tcp Active Opens. passive/s The number of times TCP connections have made a direct transition to the SYN-RCVD state from the LISTEN state per second tcp Passive Opens. iseg/s The total number of segments received per second, includ- ing those received in error tcpInSegs]. This count includes segments received on currently established con- nections. […] Specific uses of sar(1) are described later in this book; see Chapters 6 to 10. Appendix C is a summary of the sar(1) options and metrics. 4.5 Tracing Tools Linux tracing tools use the previously described events interfaces (tracepoints, kprobes, uprobes, USDT) for advanced performance analysis. The main tracing tools are:

	[[perf]](1): The [[official]] [[Linux]] [[profiler]]. It is [[excellent]] for [[CPU profiling]] ([[sampling]] of [[stack trace]]s) and [[PMC]] [[analysis]], and can [[instrument]] other [[event]]s, typically [[recording]] to an [[output]] [[file]] for [[post]]-[[processing]].

	[[Ftrace]]: The [[official]] [[Linux]] [[tracer]], it is a [[multi]]-[[tool]] [[compose]]d of [[different]] [[tracing]] [[utilities]]. It is [[suite]]d for [[kernel]] [[code path]] [[analysis]] and [[resource]]-[[constrained]] [[system]]s, as it can be used without [[dependencies]].

	[[BPF]] ([[BCC]], [[bpftrace]]): [[Extended BPF]] was [[introduce]]d in [[Chapter]] 3, [[Operating System]]s, [[Section]] 3.4.4, [[Extended BPF]]. It [[power]]s [[advanced]] [[tracing tool]]s, the [[main]] ones being [[BCC]] and [[bpftrace]]. [[BCC]] [[provide]]s [[powerful]] [[tool]]s, and [[bpftrace]] [[provide]]s a [[high-level language]] for [[custom]] one-[[liner]]s and [[short]] [[program]]s.

	[[SystemTap]]: A [[high-level language]] and [[tracer]] with many tapsets ([[libraries]]) for [[tracing]] [[different]] [[target]]s [[Eigler]] 05][[Source]][[ware]] 20]. It has recently been [[developing]] a [[BPF]] [[backend]], which I [[recommend]] (see the stapbpf(8) [[man page]]).

	LTTng: A [[tracer]] [[optimized]] for [[black]]-box]] [[recording]]: [[optimal]]ly [[recording]] many [[event]]s for later [[analysis]] [LTTng 20].

The first three tracers are covered in Chapter 13, perf; Chapter 14, Ftrace; and Chapter 15, BPF. The chapters that now follow (5 to 12) include various uses of these tracers, showing the commands to type and how to interpret the output. This ordering is deliberate, focusing on uses and performance wins first, and then covering the tracers in more detail later if and as needed. At Netflix, I use perf(1) for CPU analysis, Ftrace for kernel code digging, and BCC/bpftrace for everything else (memory, file systems, disks, networking, and application tracing). 4.6 Observing Observability Observability tools and the statistics upon which they are built are implemented in software, and all software has the potential for bugs. The same is true for the documentation that describes the software. Regard with a healthy skepticism any statistics that are new to you, questioning what they really mean and whether they are really correct. Metrics may be subject to any of the following problems:

	[[Tool]]s and [[measurement]]s are some[[time]]s wrong.

	[[Man page]]s are not always [[right]].

	[[Available]] [[metric]]s may be [[incomplete]].

	[[Available]] [[metric]]s may be poorly [[designed]] and [[conf]][[using]].

	[[Metric]] [[collector]]s (e.g., that [[parse]] [[tool]] [[output]]) can have [[bug]]s.13

13In this case the tool and measurement are correct, but an automated collector has introduced errors. At Surge 2013 I gave a lightning talk on an astonishing case Gregg 13c]: a benchmarking company report ed poor metrics for a product I was supporting, and I dug in. It turned out the shell script they used to automate the benchmark had two bugs. First, when processing output from fio(1), it would take a result such as “100KB/s” and use a regular expression]] to elide nun-numeric characters, including “KB/s]]” to turn this into “100”. Since fio(1) report ed results with different units (bytes, Kbytes, Mbytes), this introduced massive (1024x) errors. Second, they also elided decimal places, so a result of “1.6” became “16”.

	[[Metric]] [[processing]] ([[algorithm]]s/[[spreadsheet]]s) can also [[introduce]] [[error]]s.

When multiple observability tools have overlapping coverage, you can use them to cross-check each other. Ideally, they will source different instrumentation frameworks to check for bugs in the frameworks as well. Dynamic instrumentation is especially useful for this purpose, as custom tools can be created to double-check metrics. Another verification technique is to apply known workloads and then to check that the observability tools agree with the results you expect. This can involve the use of micro-benchmarking tools that report their own statistics for comparison. Sometimes it isn’t the tool or statistic that is in error, but the documentation that describes it, including man pages. The software may have evolved without the documentation being updated. Realistically, you may not have time to double-check every performance measurement you use and will do this only if you encounter unusual results or a result that is company critical. Even if you do not double-check, it can be valuable to be aware that you didn’t and that you assumed the tools were correct. Metrics can also be incomplete. When faced with a large number of tools and metrics, it may be tempting to assume that they provide complete and effective coverage. This is often not the case: metrics may have been added by programmers to debug their own code, and later built into observability tools without much study of real customer needs. Some programmers may not have added any at all to new subsystems. An absence of metrics can be more difficult to identify than the presence of poor metrics. Chapter 2, Methodologies, can help you find these missing metrics by studying the questions you need answer ed for performance analysis. 4.7 Exercises Answer the following questions about observability tools (you may wish to re visit the introduction to some of these terms in Chapter 1):

	[[List]] some [[static performance tool]]s.

	[[What is]] [[profiling]]?

	[[Why]] would [[profiler]]s [[use]] 99 [[Hertz]] instead of 100 [[Hertz]]?

	[[What is]] [[tracing]]?

	[[What is]] [[static instrumentation]]?

	[[Describe]] [[why]] [[dynamic instrumentation]] is [[important]].

	[[What is]] the [[difference]] between [[tracepoint]]s and [[kprobe]]s?

	[[Describe]] the [[expected]] [[CPU]] [[overhead]] ([[low]]/medium/[[high]]) from the [[follow]]ing:

	[[Disk]] [[IOPS]] [[counter]]s (as seen by [[iostat]](1))

	[[Tracing]] per-[[event]] [[disk I/O]] via [[tracepoint]]s or [[kprobe]]s

	[[Tracing]] per-[[event]] [[context]] [[switch]]es ([[tracepoint]]s/[[kprobe]]s)

	[[Tracing]] per-[[event]] [[process]] [[execution]] ([[exec]]ve(2)) ([[tracepoint]]s/[[kprobe]]s)

	[[Tracing]] per-[[event]] [[libc]] [[malloc]]() via [[uprobe]]s

	[[Describe]] [[why]] [[PMC]]s are [[valuable]] for [[performance analysis]].

	Given an [[observability tool]], [[describe]] how you could determine what [[instrumentation]] [[source]]s it [[use]]s.

4.8 References Eigler 05] Eigler, F. Ch., et al. “Architecture of SystemTap: A Linux Trace/Probe Tool,” http://source ware.org/systemtap/arch paper.pdf, 2005. [Drongowski 07] Drongowski, P., “Instruction-Based Sampling: A New Performance Analysis Technique for AMD Family 10h Processors,” AMD (White paper), 2007. [Ayuso 12] Ayuso, P., “The Conn[[track-Tools User Manual,” http://conn[[track-tools.net[[filter.org/manual.html, 2012. Gregg 13c] Gregg, B., “Benchmarking Gone Wrong,” Surge 2013: Lightning Talks, https://www.youtube.com/watch?v=vm1GJMp0QN4#t=17m48s, 2013. [Weisbecker 13] Weisbecker, F., “Status of Linux dynticks,” OSPERT, http://www.ertl.jp/~shinpei/conf/ospert13/slides/FredericWeisbecker.pdf, 2013. Intel 16] Intel 64 and IA-32 Architectures Software Developer’s Manual Volume 3B: System Programming Guide, Part 2, September 2016, https://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-software-developer-vol-3b-part-2-manual.html, 2016. AMD 18] Open-Source Register Reference for AMD Family 17h Processors Models 00h-2Fh, https://developer.amd.com/resources/developer-guides-manuals, 2018. ARM 19] Arm® Architecture Reference Manual Armv8, for Armv8-A architecture profile, https://developer.arm.com/architectures/cpu-architecture/a-profile/docs?_ga=2.78191124.1893781712.1575908489-930650904.1559325573, 2019. Gregg 19] Gregg, B., BPF Performance Tools: Linux System and Application Observability, Addison-Wesley, 2019. [Atoptool 20] “Atop,” www.atop tool.nl/index.php, accessed 2020. [Bowden 20] Bowden, T., Bauer, B., et al., “The /proc Filesystem,” Linux documentation, https://www.kernel.org/doc/html/latest/filesystems/proc.html, accessed 2020. Gregg 20a] Gregg, B., “Linux Performance,” http://www.brendangregg.com/linux perf.html, accessed 2020. [LTTng 20] “LTTng,” https://lttng.org, accessed 2020. PCP 20] “Performance Co-Pilot,” https://pcp.io, accessed 2020. [Prometheus 20] “Exporters and Integrations,” https://prometheus.io/docs/instrumenting]]/exporters, accessed 2020. Source ware 20] “SystemTap,” https://source ware.org/systemtap, accessed 2020. Ts’o 20] Ts’o, T., Zefan, L., and Zanussi, T., “Event Tracing,” Linux documentation, https://www.kernel.org/doc/html/latest/trace/events.html, accessed 2020. [Xenbits 20] “Xen Hypervisor Command Line Options,” https://xenbits.xen.org/docs/4.11-testing/misc/xen-command-line.html, accessed 2020. [UTK 20] “Performance Application Programming Interface,” http://icl.cs.utk.edu/papi, accessed 2020. ==Fair Use Sources== Fair Use Sources:

B08J5QZPNC (SysPrfBGrg 2021)

Systems Performance Observability Tools