Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix tests broken by refactor

    • -9
    • +8
    ./CrossVersionPerformanceTestRunnerTest.groovy
    • -17
    • +0
    ./GradleInternalCrossVersionPerformanceTestRunnerTest.groovy
  1. … 1 more file in changeset.
Refactor performance baseline version selection (#11015)

Previously, performance test version selection is a little mess

and versions higher than minimumBaseVersion might be executed.

This commit refactors version selection code and ignore the historical

performance test if all versions don't meet minimumBaseVersion requirement.

    • -8
    • +17
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 3 more files in changeset.
Resolve review feddback

    • -1
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Refactor version selection

Previously, performance test version selection is a little mess

and versions higher than minimumBaseVersion might be executed.

This commit refactors version selection code and ignore the historical

performance test if all versions don't meet minimumBaseVersion requirement.

    • -8
    • +18
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Refactor version selection

Previously, performance test version selection is a little mess

and versions higher than minimumBaseVersion might be executed.

This commit refactors version selection code and ignore the historical

performance test if all versions don't meet minimumBaseVersion requirement.

    • -8
    • +18
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Refactor version selection

Previously, performance test version selection is a little mess

and versions higher than minimumBaseVersion might be executed.

This commit refactors version selection code and ignore the historical

performance test if all versions don't meet minimumBaseVersion requirement.

    • -8
    • +18
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Refine performance baseline selection logic (#10972)

Refine performance baseline selection logic

See https://github.com/gradle/gradle-private/issues/2571

    • -0
    • +57
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 25 more files in changeset.
Refine performance baseline selection logic (#10972)

Refine performance baseline selection logic

See https://github.com/gradle/gradle-private/issues/2571

    • -0
    • +57
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 25 more files in changeset.
Refine performance baseline selection logic

See https://github.com/gradle/gradle-private/issues/2571

    • -0
    • +57
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 25 more files in changeset.
Rename performance test infrastructure legacy classes

To make clear that they are using the Gradle build internal

infrastructure.

    • -413
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
    • -0
    • +412
    ./GradleInternalCrossVersionPerformanceTestRunnerTest.groovy
  1. … 43 more files in changeset.
Generate gradle profiler report as well

    • -1
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 5 more files in changeset.
Generate gradle profiler report as well

    • -1
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 5 more files in changeset.
spelling: previous

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

Fail performance test iteration fast (#7777)

This fixes https://github.com/gradle/gradle-private/issues/1393

Previously, we collect all exceptions in the performance test execution then `assertEveryBuildSucceeds()`. This doesn't make any sense - we don't need to run the test any more if there's a failure. This PR fixes this issue by failing fast in performance iterations.

Also, we have 4 kinds of checks, which makes things complicated. This PR removes them and use `true/false` instead.

    • -43
    • +0
    ./CrossVersionPerformanceTestExecutionTest.groovy
    • -29
    • +5
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 15 more files in changeset.
Simplify Profiler interface

    • -2
    • +0
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 11 more files in changeset.
Replace honest profiler with JFR

JFR is available on all systems that have an Oracle JDK, Honest profiler only works on Linux.

JFR profiles CPU, memory, IO, locking and more, Honest profiler only profiles CPU.

Both of them use the same low-overhead async profiling API.

This PR also inlines the code necessary for flamegraph generation.

Users will no longer need any additional setup on their machine.

    • -3
    • +2
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 12 more files in changeset.
Remove YourkitProfiler

This profiler wasn't very useful, as it also measured warmup runs.

We don't use it on CI and locally one can use the Gradle Profiler

with the `--profile yourkit` option.

  1. … 5 more files in changeset.
Use u-test to determine performance regressions

Instead of an arbitrary threshold we now use the Mann-Whitney U test

to calculate how confident we can be in the results.

For now we'll require a 99% confidence before reporting a regression.

We can tweak that value based on whether we miss too many

regressions or report too many non-regressions.

    • -2
    • +8
    ./CrossVersionPerformanceTestExecutionTest.groovy
    • -19
    • +30
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 10 more files in changeset.
Report test results to Slack

    • -1
    • +1
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 8 more files in changeset.
Determine latest nightly while running performance test

    • -9
    • +9
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Rename methods on `ReleasedVersionDistributions`

So they match more closely what they really return.

    • -3
    • +3
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 9 more files in changeset.
Revert "Make performance tests stricter"

This lead to too many false positives. We either

need to make our infrastructure deliver more precise

results or apply better statistical analysis.

    • -1
    • +1
    ./CrossVersionPerformanceTestExecutionTest.groovy
  1. … 1 more file in changeset.
Make performance tests stricter

    • -1
    • +1
    ./CrossVersionPerformanceTestExecutionTest.groovy
  1. … 1 more file in changeset.
Explicit clean up before performance test measurements (#2640)

Previously we were using a workaround where odd runs were removed from measurements, and executed a `clean` build instead of the measured build.

Performance tests can now specify `cleanTasks`, similar to how they specify `tasksToRun`. These `cleanTasks` will be executed before each run (warm-up and measurement runs alike).

A new column is added to performance test tables to track this new information. It is a nullable column to allow for test results added by older versions of Gradle.

I've updated the task output caching tests and the Maven vs. Gradle comparisons to declare `cleanTasks` instead of the old hack with the odd-even runs.

    • -1
    • +1
    ./CrossVersionPerformanceTestExecutionTest.groovy
    • -2
    • +4
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 26 more files in changeset.
Add host name to performance test results

    • -0
    • +1
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 15 more files in changeset.
Make performance tests stricter

    • -1
    • +1
    ./CrossVersionPerformanceTestExecutionTest.groovy
  1. … 1 more file in changeset.
Improve diagnostics for failed performance tests

    • -6
    • +9
    ./CrossVersionPerformanceTestExecutionTest.groovy
  1. … 1 more file in changeset.
Lower performance regression threshold

Our performance tests are much more stable now, so we can

lower the regression threshold without introducing too many false

positives. This will allow us to spot regressions earlier.

    • -1
    • +1
    ./CrossVersionPerformanceTestExecutionTest.groovy
  1. … 1 more file in changeset.
Fail performance tests if build under test fails

    • -0
    • +59
    ./CrossVersionPerformanceTestRunnerTest.groovy
  1. … 2 more files in changeset.
Move tests to integration test source

    • -38
    • +0
    ./IdProviderAndroidStudioMockupPerformanceTest.groovy
    • -39
    • +0
    ./IdProviderCrossBuildPerformanceTes.groovy
    • -39
    • +0
    ./IdProviderCrossVersionPerformanceTest.groovy
    • -39
    • +0
    ./IdProviderGradleVsMavenPerformanceTest.groovy
    • -38
    • +0
    ./IdProviderToolingApiPerformanceTest.groovy
  1. … 5 more files in changeset.