# How to Write Metrics Timeline-Based Measurement v2 is a system for computing metrics from traces. A TBM2 metric is a Javascript function that takes a trace Model and produces Histograms. ## Coding Practices Please follow the [Catapult Javascript style guide](/docs/style-guide.md) so that the TBM2 maintainers can refactor your metric when we need to update the TBM2 API. Please write a unit test for your metric. If your metric computes information from the trace that may be of general use to other metrics or the trace viewer UI, then the TBM2 maintainers may ask for your help to generalize your innovation into a part of the Trace Model such as the [UserModel](/tracing/tracing/model/user_model/user_model.html) or [ModelHelpers](/tracing/tracing/model/helpers/chrome_browser_helper.html). Use the dev server to develop and debug your metric. * Run `./bin/run_dev_server` * Navigate to [http://localhost:8003/tracing_examples/trace_viewer.html](http://localhost:8003/tracing_examples/trace_viewer.html). * Open a trace that your metric can be computed from. * Open the Metrics side panel on the right. * Select your metric from the drop-down. * Inspect the results and change your metric if necessary. * Open different traces to explore corner cases in your metric. ## Trace Model Trace logs are JSON files produced by tracing systems in Chrome, Android, linux perf, BattOr, etc. The trace model is an object-level representation of events parsed from a trace log. The trace model contains Javascript objects representing * OS [processes](/tracing/tracing/model/process.html), [threads](/tracing/tracing/model/thread.html), * utilities for finding special processes and threads such as [ChromeBrowserHelper](/tracing/tracing/model/helpers/chrome_browser_helper.html), * synchronous [ThreadSlices](/tracing/tracing/model/thread_slice.html) and asynchronous [AsyncSlices](/tracing/tracing/model/async_slice.html), * [snapshots](/tracing/tracing/model/object_snapshot.html) of object state as it changes throughout time, * [RPCs](/tracing/tracing/model/flow_event.html), * [FrameBlameContexts](/tracing/tracing/extras/chrome/blame_context/blame_context.html), * battery [power samples](/tracing/tracing/model/power_sample.html), * synthetic higher-level abstractions representing complex sets of events such as [UserExpectations](/tracing/tracing/model/user_model/user_expectation.html), * and [more](/tracing/tracing/model/model.html)! ## Histograms A [Histogram](/tracing/tracing/value/histogram.html) is basically a common [histogram](https://en.wikipedia.org/wiki/Histogram), but with a few extra bells and whistles that are particularly useful for TBM2 metrics. * Specify units of samples and improvement direction with [Unit](/tracing/tracing/value/unit.html) * JSON serialization with asDict()/fromDict() * Build custom bin boundaries with HistogramBinBoundaries * Underflow and overflow bins for samples outside of the range of the central bins * Compute statistics such as average, stddev, sum, and percentiles * Customize which statistics are serialized with customizeSummaryOptions() * Count non-numeric samples * Store a random subset of sample values * getDifferenceSignificance() computes whether two histograms are significantly different with a Mann-Whitney U hypothesis test * addHistogram() merges two Histograms with the same units and bin boundaries But the most complex special feature of Histograms is their Diagnostics. ## Diagnostics When a metric significantly regresses, you then need to diagnose why it regressed. Diagnostics are pieces of information that metrics attach to Histograms in order help you diagnose regressions. Diagnostics may be associated either with the entire Histogram directly, or with a particular sample. Attach a Diagnostic to a Histogram: ```javascript histogram.diagnostics.set('name', diagnostic) // or histograms.addHistogram(histogram, {name: diagnostic}) ``` Attach a Diagnostic to all Histograms in a HistogramSet: ```javascript histograms.addSharedDiagnosticToAllHistograms(name, diagnostic); ``` Attach a Diagnostic to a sample: ```javascript histogram.addSample(number, {name: diagnostic}) ``` ### General Diagnostics * [GenericSet](/tracing/tracing/value/diagnostics/generic_set.html): This can contain any data that can be serialized and deserialized using JSON.stringify() and JSON.parse(), including numbers, strings, Arrays, and dictionaries (simple Objects). It will be visualized using [generic-object-view](/tracing/tracing/ui/analysis/generic_object_view.html), which is quite smart about displaying tabular data using tables, URLs using HTML anchor tags, pretty-printing, recursive data structures, and more.  * [Breakdown](/tracing/tracing/value/diagnostics/breakdown.html): Structurally, these are Maps from strings to numbers. Conceptually, they describe what fraction of a whole (either a Histogram or a sample) is due to some sort of category - either a category of event, CPU sample, memory consumer, whathaveyou. Visually, they are a stacked bar chart with a single bar, which is spiritually a pie chart, but less misleading.  * [RelatedEventSet](/tracing/tracing/value/diagnostics/related_event_set.html): This is a Set of references to Events in the trace model. Visually, they are displayed as HTML links which, when clicked in the metrics-side-panel, select the referenced Events in the trace viewer's timeline view. When clicked in results.html, they currently do nothing, but should eventually open the trace that contains the events and select them.  * [DateRange](/tracing/tracing/value/diagnostics/date_range.html): This is a Range of Dates. It cannot be empty, but the minDate could be the same as the maxDate. Telemetry automatically adds 2 shared DateRanges to all results: 'benchmarkStart' and 'traceStart'.  ### Histogram Relationship Diagnostics * [RelatedNameMap](/tracing/tracing/value/diagnostics/related_name_map.html): This maps from short keys to Histogram name. These are correlated with Breakdowns. They are visualized as HTML links in Breakdowns. ### Other Diagnostics * [Scalar](/tracing/tracing/value/diagnostics/scalar.html): Metrics must not use this, since it is incapable of being merged. It is mentioned here for completeness. It wraps a Scalar, which is just a unitted number. This is only to allow Histograms in other parts of the trace viewer to display number sample diagnostics more intelligently than GenericSet can. If a metric wants to display number sample diagnostics intelligently, then it should use Breakdown and RelatedNameMap. ### Reserved Names Metrics may not use the following names for Histogram-level Diagnostics. * alertGrouping is a GenericSet of strings. Each string is a grouping key that will be used by the dashboard auto-triage system to [group alerts](https://goto.google.com/chromeperf-alert-grouping-dd) together. * angleRevisions is a GenericSet of strings containing [Angle](https://chromium.googlesource.com/angle/angle/) git hashes. * architectures is a GenericSet of strings containing [CPU architectures](https://en.wikipedia.org/wiki/List_of_CPU_architectures). * benchmarks is a GenericSet of strings containing Telemetry benchmark names. * benchmarkStart is a DateRange containing timestamps of Telemetry benchmark runs. * bots is a GenericSet of strings containing bot hostnames. * bugComponents is a GenericSet of strings containing [Monorail components](https://bugs.chromium.org/p/chromium/issues/advsearch). * builds is a GenericSet of numbers containing Chromium build numbers. * catapultRevisions is a GenericSet of strings containing [Catapult](https://chromium.googlesource.com/catapult) git hashes. * chromiumCommitPositions is a GenericSet of numbers containing Chromium commit positions. * chromiumRevisions is a GenericSet of strings containing [Chromium](https://chromium.googlesource.com/chromium/src/) git hashes. * documentationUrls is a GenericSet of strings containing the urls to the documentation of the benchmarks/metrics. * gpus is a GenericSet of objects containing metadata about GPUs. * labels is a GenericSet of strings containing [user-defined labels](https://chromium.googlesource.com/catapult/+/b0f1e24d4686b3ce46667c0124a186e414fbd006/telemetry/telemetry/internal/results/results_options.py#82) for Telemetry results. * logUrls is a GenericSet of strings containing URLs pointing to human-readable logs. * masters is a GenericSet of strings containing bot master hostnames. * memoryAmounts is a GenericSet of numbers containing the total amount of RAM in the device that recorded the Chromium trace. * osNames is a GenericSet of strings containing names of OSs like 'linux' and 'mac'. * osVersions is a GenericSet of strings containing OS versions. * owners is a GenericSet of strings containing email addresses of owners of Telemetry benchmarks. * productVersions is a GenericSet of strings containing Chromium product versions like '60.0.9999.0'. * relatedNames is a GenericSet of strings containing names of related Histograms. * skiaRevisions is a GenericSet of strings containing [Skia](https://chromium.googlesource.com/skia/) git hashes. * stories is a GenericSet of strings containing Telemetry story names. * storysetRepeats is a GenericSet of numbers containing Telemetry storyset repetition counters. * storyTags is a GenericSet of strings containing Telemetry story tags. * tagmap maps from Telemetry story tags to story names. * traceStart is a DateRange containing timestamps of Chromium traces. * traceUrls is a GenericSet of strings containing URLs pointing to Chromium traces. * v8CommitPositions is a GenericSet of numbers containing V8 commit positions. * v8Revisions is a GenericSet of strings containing [V8](https://chromium.googlesource.com/v8/v8/) git hashes. * webrtcRevisions is a GenericSet of strings containing [webrtc](https://chromium.googlesource.com/external/webrtc/) git hashes. Consumers of metrics results generally cannot rely on the presence or absence of any of these metadata diagnostics. Consumers can rely on the presence of Telemetry metadata for results produced by Telemetry. If present, they may contain a single value or multiple values. If uploading to the ChromePerf dashboard, the following diagnostics are required to be shared by all Histograms, and must all contain exactly one value: * bots * benchmarks * chromiumCommitPositions * masters ## Alert grouping Setting alert groupings for your histograms allows the dashboard to be [smarter](https://goto.google.com/chromeperf-alert-grouping-dd) about how to group alerts together. In order to define alert groupings for your metrics, first add it to `tracing/tracing/value/diagnostics/alert_groups.html`. Then you can set these alert groupings either by calling `setAlertGrouping` on your histgoram: ```javascript my_hist.setAlertGrouping([ tr.v.d.ALERT_GROUPS.LOADING_PAINT, tr.v.d.ALERT_GROUPS.LOADING_INTERACTIVITY]); ``` or by defining it when you create your histogram: ```javascript const my_hist = tr.v.Histogram.create('foo', unitlessNumber, [], { alertGrouping: [ tr.v.d.ALERT_GROUPS.LOADING_PAINT, tr.v.d.ALERT_GROUPS.LOADING_INTERACTIVITY, ]}); ``` ## Consumers of Histograms Histograms are consumed by * [histogram-set-table](/tracing/tracing/value/ui/histogram_set_table.html) in both results.html and the Metrics side panel in trace viewer, * the [dashboard](https://chromeperf.appspot.com) indirectly via their statistics. Currently, telemetry discards Histograms and Diagnostics, and only passes their statistics scalars to the dashboard. Histograms and their Diagnostics will be passed directly to the dashboard early 2017. Metrics can control which statistics are uploaded to the dashboard by passing a dictionary to customizeSummaryOptions() to enable or disable statistics. The default options are as follows: * `avg` (average/mean): true * `geometricMean`: false * `std` (standard deviation): true * `count` (number of samples): true * `sum`: true * `min`: true * `max`: true * `nans` (number of non-numeric samples): false * `percentile`: [] * Unlike the other options which are booleans, percentile is an array of numbers between 0 and 1. In order to upload the median, for example, a metric would call `histogram.customizeSummaryOptions({percentile: [0.5]})`. ## How histogram-set-table Uses Merging The histogram-set-table element uses the predefined [HistogramGroupings](/tracing/tracing/value/histogram_set.html), along with the merging capabilities of Histograms, to allow dynamic, hierarchical organization of histograms: * Predefined HistogramGroupings specify how to find the benchmark, story, etc. that produced the Histogram. * After loading histograms, histogram-set-table computes categories to be displayed by the groupby picker at the top of the UI: * Categories are HistogramGroupings that have more than one value across all histograms in the HistogramSet. * Instead of having one category for all story grouping keys, each grouping individual grouping key may be listed as a category. For example, in Page Cycler v2 benchmarks, the "cache_temperature" grouping key would be displayed as a category. * Choosing groups builds a hierarchy of histograms that is filled in by merging histograms from the bottom up. Expanding the rows of histogram-set-table, any leaf nodes are histograms that were loaded, and their ancestors are computed by merging. * histogram-set-table uses the "label" HistogramGrouping to define the columns of the table.