CrossVersionPerformanceTestRunnerTest.groovy

Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Refine performance baseline selection logic

See https://github.com/gradle/gradle-private/issues/2571

    • -0
    • +57
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 25 more files in changeset.
Rename performance test infrastructure legacy classes

To make clear that they are using the Gradle build internal

infrastructure.

    • -413
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 44 more files in changeset.
Generate gradle profiler report as well

    • -1
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 5 more files in changeset.
Generate gradle profiler report as well

    • -1
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 5 more files in changeset.
Fail performance test iteration fast (#7777)

This fixes https://github.com/gradle/gradle-private/issues/1393

Previously, we collect all exceptions in the performance test execution then `assertEveryBuildSucceeds()`. This doesn't make any sense - we don't need to run the test any more if there's a failure. This PR fixes this issue by failing fast in performance iterations.

Also, we have 4 kinds of checks, which makes things complicated. This PR removes them and use `true/false` instead.

    • -29
    • +5
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 16 more files in changeset.
Simplify Profiler interface

    • -2
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 11 more files in changeset.
Replace honest profiler with JFR

JFR is available on all systems that have an Oracle JDK, Honest profiler only works on Linux.

JFR profiles CPU, memory, IO, locking and more, Honest profiler only profiles CPU.

Both of them use the same low-overhead async profiling API.

This PR also inlines the code necessary for flamegraph generation.

Users will no longer need any additional setup on their machine.

    • -3
    • +2
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 12 more files in changeset.
Use u-test to determine performance regressions

Instead of an arbitrary threshold we now use the Mann-Whitney U test

to calculate how confident we can be in the results.

For now we'll require a 99% confidence before reporting a regression.

We can tweak that value based on whether we miss too many

regressions or report too many non-regressions.

    • -19
    • +30
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 11 more files in changeset.
Report test results to Slack

    • -1
    • +1
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 8 more files in changeset.
Determine latest nightly while running performance test

    • -9
    • +9
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Rename methods on `ReleasedVersionDistributions`

So they match more closely what they really return.

    • -3
    • +3
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 9 more files in changeset.
Explicit clean up before performance test measurements (#2640)

Previously we were using a workaround where odd runs were removed from measurements, and executed a `clean` build instead of the measured build.

Performance tests can now specify `cleanTasks`, similar to how they specify `tasksToRun`. These `cleanTasks` will be executed before each run (warm-up and measurement runs alike).

A new column is added to performance test tables to track this new information. It is a nullable column to allow for test results added by older versions of Gradle.

I've updated the task output caching tests and the Maven vs. Gradle comparisons to declare `cleanTasks` instead of the old hack with the odd-even runs.

    • -2
    • +4
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 27 more files in changeset.
Add host name to performance test results

    • -0
    • +1
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 15 more files in changeset.
Fail performance tests if build under test fails

    • -0
    • +59
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Remove memory measurement APIs

    • -19
    • +18
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 10 more files in changeset.
Simplify performance measurements

The many measurements that we injected into the build under test were

skewing our measurements to the point of making them unreliable or

unrealistic. We now only measure end to end build time. Here's a breakdown

with the rationale for removing each other measurement:

- configuration time: can be done by having a `gradle help` scenario instead

- execution time: the user does not care whether a long build is stuck in execution or configuration

- setup/teardown: was ill-defined anyway, basically total - configuration - execution

- JIT compile time: this is nothing we can influence and thus pointless to measure

- Memory usage: Was only measured at one point in the build, which doesn't tell us anything about

any problems at any other point in the build

- GC CPU time: If this increases we'd see it in total execution time

Generally, looking at the graphs has never pointed us directly at the problem, we always need to

profile anyway. So instead of skewing our measurements with lots of profling code, we should

instead use a dedicated profiling job to measure if we actually see a regression.

Memory usage can be tested indirectly by giving each scenario a reasonable amount of memory.

If memory usage rises above that reasonable limit, we'd see execution time rise, telling us about

the regression. Generally, we do not optimize for smallest memory usage, but for fastest execution

with reasonable memory overhead.

This change also removes all JVM tweaking and wait periods which we introduced in an attempt to

make tests more predictable and stable. These tweaks have not really helped us achieve more stable

tests and have often done the opposite. They also add lots of complexity and make our tests more

unrealistic. A real users will not add all these JVM options to Gradle.

    • -62
    • +1
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 59 more files in changeset.
Use baseVersion consistently

Note that versionBase from the build-receipt.properties is

only used in `incoming-distributions.gradle`.

PR #798

    • -4
    • +4
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 6 more files in changeset.
Set JVM options for daemon client in performance tests

- optimize for stable results

    • -1
    • +2
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Wire integration test build context instance

- enables using performance test specific build context when an instance

is properly wired

    • -2
    • +3
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 48 more files in changeset.
Make '--baselines none' work again

    • -3
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 1 more file in changeset.
Ignored a couple of tests.

    • -0
    • +3
    ./CrossVersionPerformanceTestRunnerTest.groovy
Make it possible to generate flame graphs during performance tests

This commit improves the integration of Honest Profiler by making it possible to post-process the log files and generate a flame graph.

For this to be possible, the `org.gradle.performance.honestprofiler` system property must be set to a directory where flame graphs will

be exported.

It is expected to find 2 environment variables:

- `HP_HOME_DIR` must point to a valid Honest Profiler installation (https://github.com/RichardWarburton/honest-profiler)

- `FG_HOME_DIR` must point to a valid FlameGraph installation (https://github.com/brendangregg/FlameGraph)

In case those are not set, the collector will try to find them in `~/tools/honest-profiler` and `~/tools/FlameGraph` respectively.

The graphs are *not* integrated into the performance reports yet.

    • -0
    • +2
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 3 more files in changeset.
Make it possible to generate flame graphs during performance tests

This commit improves the integration of Honest Profiler by making it possible to post-process the log files and generate a flame graph.

For this to be possible, the `org.gradle.performance.honestprofiler` system property must be set to a directory where flame graphs will

be exported.

It is expected to find 2 environment variables:

- `HP_HOME_DIR` must point to a valid Honest Profiler installation (https://github.com/RichardWarburton/honest-profiler)

- `FG_HOME_DIR` must point to a valid FlameGraph installation (https://github.com/brendangregg/FlameGraph)

In case those are not set, the collector will try to find them in `~/tools/honest-profiler` and `~/tools/FlameGraph` respectively.

The graphs are *not* integrated into the performance reports yet.

    • -0
    • +2
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 3 more files in changeset.
Polish CrossVersionPerformanceTestRunner

    • -8
    • +12
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 1 more file in changeset.
Add support for --baselines none

    • -0
    • +32
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 1 more file in changeset.
Fix tests

    • -14
    • +14
    ./CrossVersionPerformanceTestRunnerTest.groovy
Allow leaving targetVersions unspecified

- it's faster to do profiling when running only the current version

    • -0
    • +18
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 1 more file in changeset.
Split test so that each perf run has a different directory

    • -13
    • +49
    ./CrossVersionPerformanceTestRunnerTest.groovy
Reintroduce gradle/gradle@784d747

- There was previously a concern that this would break the

BuildScansPerformanceTest, but that test doesn't actually use the

BaselineVersion class at all.

    • -4
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 31 more files in changeset.
Revert "Make strict performance testing default."

This reverts commit 784d7476556e63c8d7f15919ae9875c3b7181aa3.

    • -0
    • +4
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 31 more files in changeset.