Performance insights is a new feature in Android vitals designed specifically for game developers and other developers with apps that use native code. Performance insights is powered by Android Performance Tuner, a new plug-in to supercharge Android vitals for game developers.
This article provides information on the core concepts behind Android Performance Tuner and your performance insights, and the calculations that underpin them:
- Fidelity parameters and quality levels
- Annotations
- Target frame rate and target frame time
- Slow and fast frame thresholds
- Measured frame time
- Slow and fast frame metrics
- Issues and opportunities
Core concepts
Fidelity parameters and quality levels
Your app’s fidelity parameters influence the performance and graphical fidelity of your game, such as mesh level-of-detail, texture resolution, and anti-aliasing method. They can be continuous (e.g., 0-100%), binary (yes/no), or some other set of values.
A quality level is a predefined set of values for your fidelity parameters. Quality levels enable you to define combinations of parameters that deliver an acceptable level of fidelity at a stable frame rate for some subset of your users. Not all possible combinations of parameters would deliver a meaningful user experience.
How fidelity parameters and quality levels are setYour app’s fidelity parameters and its quality levels are defined during integration of Android Performance Tuner. You can define up to 15 quality levels as part of integration, with an unlimited number of fidelity parameters. Levels are sequenced in increasing fidelity order, where 1 is the lowest fidelity level.
Your quality levels and fidelity parameters can be viewed in Play Console (Quality > Android Vitals > Overview > select Inspect quality levels), but are read-only. To change parameters you need to release an update; go to the relevant integration guide for more information.
Quality levels are used to tag sessions so that you can determine if device models are running on a quality level that is too high or too low.
Fidelity parameters are used to identify how to map each user device to a quality level.
Important: If your fidelity parameters (and quality levels) do not accurately reflect the settings of your app on user devices, then your sessions cannot be mapped to one of your pre-defined quality levels. Such sessions are shown as being on an "unknown" quality level. It’s difficult to troubleshoot issues on "unknown" quality levels, as they could contain many different levels of fidelity. We recommend reviewing your integration if this is happening.
For more information on quality levels, including user-determined quality levels, go to Troubleshoot Android Performance Tuner issues and FAQs.
Annotations
Annotations are "tags" on scenes in your game or app. They provide contextual information about what your game is doing in a given frame. Annotations are structured as:
- Annotation type: Annotation value
- For example: “Character”: “Protagonist”
Note: One frame can have multiple annotations.
How annotations are setAnnotations are defined during integration. To change annotations you need to release an update; go to the relevant integration guide section for more information.
How annotations are used
Annotations help you understand if specific scenes in your game are causing issues or have opportunities. To get the right level of granularity for issues, you need to define sufficiently granular annotations.
Target frame rate and target frame time
Your target frame rate is used to benchmark your frame time performance across all device models. To do this, it’s converted to a target frame time, and a tolerance is applied to define your slow frame threshold.
Frame rate is a metric with the unit "frames per second" (fps). Frame time is measured in "ms". A simple formula connects the two:
- frame time (ms) = 1000/frame rate (fps)
Currently you can select one of the following target frame rates:
|
Target frame rate (fps) |
Target frame time (ms) |
|
30 |
33.333 |
|
60 |
16.667 |
|
120 |
8.333 |
If you’re not sure which target frame rate to choose, start with 30fps. This is a good base frame rate for many games. You can change your target frame rate at any time. Note that this doesn’t change your underlying data, only your insights and metrics.
Slow and fast frame thresholds
Thresholds for defining insights and frame metrics are derived from your target frame rate:
- Slow frame threshold = target frame time + a tolerance
- Fast frame threshold = 25% faster than target frame time (provides some headroom)
For a given target frame rate, the thresholds are as follows:
|
Target frame rate (fps) |
Target frame time (ms) |
Slow frame threshold (ms) (SFT) |
Fast frame threshold (ms) (FFT) |
|
30 |
33.333 |
35.37 |
25.0 |
|
60 |
16.667 |
18.54 |
12.5 |
|
120 |
8.333 |
10.12 |
6.25 |
In order to provide stable and actionable insights, a tolerance is applied to the frame time to adjust for measurement inaccuracy and ensure that we surface only issues that are user-visible, rather than deviations that will not be perceptible to the user.
A 25% headroom is used to define an opportunity, but should be treated as a guide to possible opportunities, not a guarantee. You are always recommended to dive deeper and run tests before moving devices up a quality level: even when a device has headroom, it might not have enough headroom to run the next quality level at a consistent frame rate.
Android Performance Tuner calculations
Understanding the calculations that underpin the performance tuner will help you make the most of your insights.
Measured frame time
When aggregating frame time across many frames for your app, we report on 90th percentile frame time. This is defined to be the maximum frame time for the fastest 90% of your frames.
The threshold of 90% is currently a fixed threshold and cannot be changed.
The benefits of percentile over average
Good user experience requires a stable, consistent frame rate across the whole session. An average does not provide any insight into consistency, since fast and slow frames can offset each other. Using 90th percentile provides a metric that reflects how the vast majority of frames have behaved.
Frame time bucketingFrame times are collected and reported in histogram buckets, not a continuous variable, to minimize the amount of data sent from user devices. The frame time that is reported is the upper bound of the bucket in which the frame time was classified.
As a result, your reported frame times are clustered on specific values, not continuous.
Default buckets are configured to deliver good granularity for trouble-shooting on 30, 60, and 120fps targets. They are also narrow enough to ensure that your slow frames are never overstated.
Slow and fast frame metrics
Individual frames are counted as slow or fast by comparing with the frame thresholds:
- A single frame is slow if its frame time is greater than the slow frame threshold (SFT)
- A single frame is fast if its frame time is quicker than the fast frame threshold (FFT)
Frame metrics are usually calculated only in the context of a specific issue or opportunities. Examples:
- # and % slow frames on device model X on quality level 4
- # and % fast frames on annotation Y on all quality levels
- % slow and fast frames on device spec Z
Note: The one exception to this is the slow frame metric shown in the Overview summary, which is measured across all frames.
AsymmetrySlow and fast frames are not the inverse of each other: they have different thresholds and different criteria. This means that an underperforming device model or annotation can still have fast frames, and conversely an overperforming device model or annotation can still have slow frames:
|
Criteria |
Slow frames (%) |
Fast frames (%) |
|
|
Issue |
90% frame time > SFT |
> 10% by definition |
anywhere from 0-90% |
|
Opportunity |
90% frame time < FFT |
anywhere from 0-10% |
> 90% by definition |
Note: Sometimes both slow and fast frame metrics are given to provide more context on the issue or opportunity.
Issues and opportunities
Issues and opportunities enable you to take action on your frame time performance. An issue occurs if a device model or annotation (as defined above) does not consistently achieve your frame time target. An opportunity is a device model or annotation which consistently exceeds your frame time target by a sufficient margin to allow headroom for increasing fidelity.
To identify issues and opportunities, we compare your 90% percentile frame times with thresholds for slow frames (SFT) and fast frames (FFT). These thresholds are derived from your target frame time:
|
Types of insight |
Definition |
How it’s calculated |
|
Issue:
|
“at least 10% of frames are not meeting your frame time target” |
90% percentile frame time > SFT |
|
Opportunity:
|
“at least 90% of frames are significantly faster than your frame time target” |
90% percentile frame time < FFT |
Issues and opportunities are only defined in the context of:
- Device model x quality level
- Annotation x quality level
This gives you a way to investigate the issue or opportunity. For example:
- Issue on device model X on quality level 4 => look into device model X
- Opportunity on annotation Y on all quality levels => look into annotation Y
Issues and opportunities are not opposites. Issues have a lower bar to surfacing than opportunities, and there is a wide gap between a device model or annotation not having an issue, and it having an opportunity.
An annotation can be an issue on some levels and an opportunity on others. This is also true for device models, although the scenario where a device model appears on multiple QLs is an edge case.
Issues and opportunities are tied to target frame rate. If this changes, so will your issues and opportunities. However your underlying frame time data does not change.