MaxWorkersTest.groovy

Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Remove resource deadlock detection from build operation infrastructure, as it makes assumptions that are no longer true and does not belong at this level.

  1. … 9 more files in changeset.
Rationalise handling of “current” build operation and build operation ID

For an upcoming change to emit console logging as build operation progress events, we need to associate all progress logging with the build operation. This pulled a thread on some long overdue cleanup.

The end result is:

1. Base build operation infrastructure is consolidated org.gradle.internal.operations.

2. Mechanism for passing thread global current build operation is more test friendly, and better named.

3. A consistent mechanism is used for binding the current operation to the thread, instead of two mechanisms.

4. Build operation IDs are typed to OperationIdentifier.

There is no public API or user behaviour change.

  1. … 147 more files in changeset.
Simplify time handling internally and for build scans (#2857)

* Don't make TimeProvider Serializable.

This isn't safe and generally doesn't make sense.

* Extract and promote the concept of a build timer.

This was previously not well defined and being overlaid with the concept of when a user/tool requested something, which is not always the same thing.

* Pare down the deprecated org.gradle.util.Clock down to the minimum required.

Internal usage is replaced by a `getStartTime()` directly on BuildRequestContext.

What is left is only kept for backwards compatibility with scans.

* Rename TimeProvider to Clock.

* Move BuildExecutionTimer out of baseServices into core, and into a better package.

* Remove unused.

* Simplify the time package by merging types.

* Prevent the client's build started timestamp from being later than when the provider received the build request.

* Provide a dedicated mechanism for conveying the build start time to build scans.

* Consolidate the ways of formatting durations.

  1. … 179 more files in changeset.
Build operation queue wait thread can execute operations

This fixes an issue where when multiple tasks are running

concurrently and submitting work to the build operation queue,

performance can actually be worse worker leases and the

executor thread pool compete with each other. This fixes the

issue by allowing the thread waiting for work to complete to

also execute some of that work.

  1. … 2 more files in changeset.
Introduce BuildOperationIdFactory in global scope service for unique ids

This ensures that IDs are unique among all nested builds of

one execution.

#2622

  1. … 11 more files in changeset.
Create a test fixture for ParallelismConfigurationManager

  1. … 29 more files in changeset.
Rename ParallelExecutionManager to ParallelismConfigurationManager

  1. … 35 more files in changeset.
Merge branch 'release'

# Conflicts:

# subprojects/base-services/src/test/groovy/org/gradle/internal/operations/DefaultBuildOperationExecutorParallelExecutionTest.groovy

# subprojects/base-services/src/test/groovy/org/gradle/internal/operations/MaxWorkersTest.groovy

# subprojects/core/src/main/java/org/gradle/internal/progress/DefaultBuildOperationExecutor.java

# subprojects/core/src/main/java/org/gradle/internal/service/scopes/BuildSessionScopeServices.java

# subprojects/core/src/main/java/org/gradle/testfixtures/internal/TestGlobalScopeServices.java

# subprojects/core/src/test/groovy/org/gradle/internal/progress/DefaultBuildOperationExecutorTest.groovy

# subprojects/docs/src/docs/release/notes.md

# subprojects/platform-native/src/test/groovy/org/gradle/nativeplatform/toolchain/internal/NativeCompilerTest.groovy

# subprojects/testing-jvm/src/test/groovy/org/gradle/api/internal/tasks/testing/junit/report/DefaultTestReportTest.groovy

# subprojects/testing-jvm/src/test/groovy/org/gradle/api/internal/tasks/testing/junit/result/Binary2JUnitXmlReportGeneratorSpec.groovy

  1. … 10 more files in changeset.
Fail when parallel build operations are started inside a resource lock transform

  1. … 11 more files in changeset.
Basic support for parallelism configuration notifications

  1. … 49 more files in changeset.
Group logs by project and task

Introduces a BuildOperationType enum and adds it to the

default BuildOperationDescriptor. Passes this information

to the logging infrastructure through the BuildOperationExecutor.

All BuildOperations now have a ProgressLogger. This allows

progress logging to maintain a heirarchy of build operations,

otherwise we would not be able to group many log events.

ProgressStartEvents are given additional information in a

compact form to allow a tree of build operations to be

maintained and their operation types associated. This allows us

to associate log events who aren't fired by progress logging to

still be grouped.

A LogGroupingOutputEventListener uses these build operation

types and the build operation heirarchy to buffer and output

logs related to tasks and project configurations.

Issue: #1818

  1. … 30 more files in changeset.
Add build operation types to log events

This will allow OutputEventListeners to react to output events

from different types of operations differently.

Remove BuildProgressLogger and move progress bar logic and

rendering to BuildStatusRenderer where it belongs.

Issue: #1818

  1. … 46 more files in changeset.
Unify BuildOperationExecutor and BuildOperationProcessor APIs

This introduces the following `BuildOperationExecutor`

interface (as outlined in gradle/gradle#1676):

void run(RunnableBuildOperation buildOperation);

<T> T call(CallableBuildOperation<T> buildOperation);

<O extends RunnableBuildOperation> void runAll(

Action<BuildOperationQueue<O>> schedulingAction);

<O extends BuildOperation> void runAll(

BuildOperationWorker<O> worker,

Action<BuildOperationQueue<O>> schedulingAction);

To accomplish this, the following changes were performed:

- Various representation of build operations have been merged into

1) BuildOperation (with sub-interfaces)

-> declare and describe a build operation

2) BuildOperationDescriptor (BuildOperationDescriptor.Builder)

-> describe a build operation

3) BuildOperationState

-> represents a running build operation, with run state, start time,

parent relationship information, and description

- The DefaultBuildOperationExecutor and DefaultBuildOperationProcessor

implementations have been merged in DefaultBuildOperationExecutor,

which is now build session scoped.

  1. … 180 more files in changeset.
Revert "Allow nested usage of BuildOperationProcessor"

This reverts commit 32b197058db9a184ef8d97c99104697010b9f82f.

This reverts commit f38397635c037dd0f6e0cff326538593d66b9a56.

This reverts commit 415d784af03e1d473bcac7a3309519ca22794d20.

  1. … 6 more files in changeset.
Rework MaxWorkersTest to fix flakiness

“BuildOperationWorkerRegistry operations nested in BuildOperationProcessor operations borrow parent lease”

We are not interested in the order of execution here, only in that

the two executions do not overlap.

Follow up to 32b1970

Allow nested usage of BuildOperationProcessor

Use a dynamically sized thread pool in BuildOperationProcessor to allow

nested usage. Maximum in-flight work is managed by worker leases, the

size of that thread pool has nothing to do with that. It being caped

at maxWorkers made it impossible to use BuildOperationProcessor inside

a work enqueued to BuildOperationProcessor up to a certain degree. This

commit fixes that.

  1. … 6 more files in changeset.
Revert "Allow `BuildOperationProcessor` to be used without parent operation"

This reverts commit da1ef0cbad0f7b021f39b9fba6a3484248a4c23d.

The workaround introduced in the reverted commit is not necessary anymore.

  1. … 10 more files in changeset.
BuildOperationProcessor emits build operation events

WIP

  1. … 23 more files in changeset.
Introduce a resource lock coordination service

- convert worker leases to resource locks

- atomically lock project and worker lease when selecting a task to execute

- change task execution plan to lock around resource lock state

  1. … 75 more files in changeset.
Introduce a resource lock coordination service

- convert worker leases to resource locks

- atomically lock project and worker lease when selecting a task to execute

- change task execution plan to lock around resource lock state

  1. … 76 more files in changeset.
Combine project lock service with build operation worker registry

  1. … 28 more files in changeset.
Use project locking to parallelize tasks with async work

- Allow tasks to start while running tasks are waiting on async work

- Lock on the entire build when --parallel is not used

- Discontinue handling of @ParallelizableTask

  1. … 40 more files in changeset.
Use project locking to parallelize tasks with async work

- Allow tasks to start while running tasks are waiting on async work

- Lock on the entire build when --parallel is not used

- Discontinue handling of @ParallelizableTask

  1. … 40 more files in changeset.
Allow `BuildOperationProcessor` to be used without parent operation

Previously, any calls to `BuildOperationProcessor` would fail if no

operation had been explicitly started via the `BuildOperationWorkerRegistry`.

With this change, an implicit parent operation is started if required

when operations are run via the `BuildOperationProcessor`.

In the long term, we want to merge `BuildOperationProcessor` with

`BuildOperationExecuter`, which will make this change largely

unnecessary.

  1. … 10 more files in changeset.
Better max workers concurrent tests

More coverage of BuildOpProcessor & BuildOpWorkerRegistry collaboration

Fix BuildOperationProcessor use of worker leases

  1. … 4 more files in changeset.
BuildOperationWorkerRegistry and BuildOperationProcessor work in concert

To honor —max-workers with both worker threads and processes

BuildOperationProcessor uses BuildOperationWorkerRegistry around

each executed operation. Its thread pool size still is max-workers.

BuildOperationWorkerRegistry untouched, manages worker leases and their

nesting.

    • -0
    • +95
    ./MaxWorkersTest.groovy
  1. … 9 more files in changeset.