AbstractGradleBuildPerformanceTestRunner.groovy

Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Separate Gradle profiler specific options

    • -26
    • +3
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 7 more files in changeset.
First working versino

    • -2
    • +22
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 12 more files in changeset.
Fix issue of worker build failing abnormally (#10249)

This fixes https://github.com/gradle/gradle-private/issues/2273

Previously we only read the test class name from JUnit xml result. When the worker build fails prematurely, the JUnit xml result is not available. This PR fixes this issue by adding test class name to scenario list.

    • -2
    • +2
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 8 more files in changeset.
Fix class name

    • -1
    • +2
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 4 more files in changeset.
Fix 404 of worker build

    • -2
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 6 more files in changeset.
Fix 404 of worker build

    • -2
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 6 more files in changeset.
Fix 404 of worker build

    • -2
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 6 more files in changeset.
Fix 404 of worker build

    • -2
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 5 more files in changeset.
Fail performance test iteration fast (#7777)

This fixes https://github.com/gradle/gradle-private/issues/1393

Previously, we collect all exceptions in the performance test execution then `assertEveryBuildSucceeds()`. This doesn't make any sense - we don't need to run the test any more if there's a failure. This PR fixes this issue by failing fast in performance iterations.

Also, we have 4 kinds of checks, which makes things complicated. This PR removes them and use `true/false` instead.

    • -1
    • +0
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 16 more files in changeset.
Add teamcity build id to performance tests (#7524)

This fixes https://github.com/gradle/gradle-private/issues/1535

Previously, we only don't have buildId recorded in performance database, which might cause incorrect result if running multiple performance builds against same commit. This PR adds `teamCityBuildId` column to performance test database.

A bonus is that we now have build url link in performance graph page.

    • -0
    • +4
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 17 more files in changeset.
Use repository mirrors in performance tests

    • -1
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 10 more files in changeset.
Simplify Profiler interface

    • -6
    • +0
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 11 more files in changeset.
Replace honest profiler with JFR

JFR is available on all systems that have an Oracle JDK, Honest profiler only works on Linux.

JFR profiles CPU, memory, IO, locking and more, Honest profiler only profiles CPU.

Both of them use the same low-overhead async profiling API.

This PR also inlines the code necessary for flamegraph generation.

Users will no longer need any additional setup on their machine.

    • -7
    • +4
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 12 more files in changeset.
Strive to keep Gradle's monotonic clock in sync with the system clock

    • -3
    • +3
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 71 more files in changeset.
Simplify time handling internally and for build scans (#2857)

* Don't make TimeProvider Serializable.

This isn't safe and generally doesn't make sense.

* Extract and promote the concept of a build timer.

This was previously not well defined and being overlaid with the concept of when a user/tool requested something, which is not always the same thing.

* Pare down the deprecated org.gradle.util.Clock down to the minimum required.

Internal usage is replaced by a `getStartTime()` directly on BuildRequestContext.

What is left is only kept for backwards compatibility with scans.

* Rename TimeProvider to Clock.

* Move BuildExecutionTimer out of baseServices into core, and into a better package.

* Remove unused.

* Simplify the time package by merging types.

* Prevent the client's build started timestamp from being later than when the provider received the build request.

* Provide a dedicated mechanism for conveying the build start time to build scans.

* Consolidate the ways of formatting durations.

    • -3
    • +3
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 179 more files in changeset.
Separate working directories in cross-build perf tests

    • -1
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 2 more files in changeset.
Simplify performance measurements

The many measurements that we injected into the build under test were

skewing our measurements to the point of making them unreliable or

unrealistic. We now only measure end to end build time. Here's a breakdown

with the rationale for removing each other measurement:

- configuration time: can be done by having a `gradle help` scenario instead

- execution time: the user does not care whether a long build is stuck in execution or configuration

- setup/teardown: was ill-defined anyway, basically total - configuration - execution

- JIT compile time: this is nothing we can influence and thus pointless to measure

- Memory usage: Was only measured at one point in the build, which doesn't tell us anything about

any problems at any other point in the build

- GC CPU time: If this increases we'd see it in total execution time

Generally, looking at the graphs has never pointed us directly at the problem, we always need to

profile anyway. So instead of skewing our measurements with lots of profling code, we should

instead use a dedicated profiling job to measure if we actually see a regression.

Memory usage can be tested indirectly by giving each scenario a reasonable amount of memory.

If memory usage rises above that reasonable limit, we'd see execution time rise, telling us about

the regression. Generally, we do not optimize for smallest memory usage, but for fastest execution

with reasonable memory overhead.

This change also removes all JVM tweaking and wait periods which we introduced in an attempt to

make tests more predictable and stable. These tweaks have not really helped us achieve more stable

tests and have often done the opposite. They also add lots of complexity and make our tests more

unrealistic. A real users will not add all these JVM options to Gradle.

    • -2
    • +2
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 59 more files in changeset.
Wire integration test build context instance

- enables using performance test specific build context when an instance

is properly wired

    • -2
    • +6
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 48 more files in changeset.
Add ForkingUnderDevelopmentGradleDistribution and use it for perf tests

- reduces differences between master vs. snapshot versions

in performance test execution

    • -2
    • +2
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 6 more files in changeset.
Fix startTime and endTime in performance tests

- broke by 1b20ea8d

    • -1
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 5 more files in changeset.
Control default JVM options for performance tests in one location

    • -9
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 3 more files in changeset.
Set -XX:+AlwaysPreTouch JVM option to allocate heap at startup time

- could possible reduce one source of flakiness in performance tests

    • -0
    • +1
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 1 more file in changeset.
Unify how JVM options are set for cross build performance tests

    • -0
    • +11
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 2 more files in changeset.
Moved `Clock` and `TimeProvider` into a separate package

This commit reverts recent changes to the API of

`org.gradle.util.Clock`, and instead deprecates the existing type

replacing with a new internal type. This was done because the

`ExtractDslMetaDataTask` uses this type for timing, and external plugins may have copied this pattern.

`org.gradle.util.Clock` is now deprecated.

    • -2
    • +2
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 103 more files in changeset.
Add TimeProvider.currentTimeForDuration and use it

There were many places we were measuring a duration by comparing

`System.currentTimeMillis()` at some start and end times. This is a

known anti-pattern which is subject to incorrect duration measurements

due to how clock skew affects the implementation of that function.

The new `currentTimeForDuration()` method uses `System.nanoTime()` which

is much more reliable for measuring durations within a single thread of

control in a single process.

Replacing our uses of `System.currentTimeMillis()` with

`currentTimeForDuration()` will make Gradle and our tests much more

reliable.

In service of issue #740

    • -1
    • +4
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 37 more files in changeset.
Support customizing invocations in performance tests

- remove previous parameterized generics that lead to a dead end

    • -1
    • +3
    ./AbstractGradleBuildPerformanceTestRunner.groovy
  1. … 13 more files in changeset.
Revert "Support generation of flame grahs on cross-build tests"

This reverts commit 55efcff0fba23da1f55f9410a9025cb9d52929ea.

    • -4
    • +0
    ./AbstractGradleBuildPerformanceTestRunner.groovy
Support generation of flame grahs on cross-build tests

    • -0
    • +4
    ./AbstractGradleBuildPerformanceTestRunner.groovy
Support generation of flame grahs on cross-build tests

    • -0
    • +4
    ./AbstractGradleBuildPerformanceTestRunner.groovy
Support generation of flame grahs on cross-build tests

    • -0
    • +4
    ./AbstractGradleBuildPerformanceTestRunner.groovy