Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Decouple `InMemoryModuleMetadataCache` from `DefaultModuleMetadataCache`

Isolate the in-memory caching from the persistent caching, so that we can

later use a different in-memory cache for all projects.

    • -38
    • +18
    ./InMemoryModuleMetadataCache.java
  1. … 2 more files in changeset.
Decouple `InMemoryModuleMetadataCache` from `DefaultModuleMetadataCache`

Isolate the in-memory caching from the persistent caching, so that we can

later use a different in-memory cache for all projects.

    • -0
    • +67
    ./AbstractModuleMetadataCache.java
    • -38
    • +18
    ./InMemoryModuleMetadataCache.java
  1. … 2 more files in changeset.
Fix "platform owners" not being serialized

    • -3
    • +49
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 5 more files in changeset.
Fix caching of component metadata rules in presence of `withDependencies`

This commit fixes several issues with component metadata rule `@CacheableRule`:

1. when using Gradle metadata, the original content hash wasn't set, leading

to incorrect keys in cache. The consequence is that we would fetch from cache

the wrong components.

2. there was a race condition between the component metadata serializer and

the graph resolution, which lead to rules being executed several times, and

actually multiple `DefaultConfigurationMetadata` created for the same components

3. a `ClassCastException` was visible as soon as one rule worked on dependencies

or dependency constraints, because we assume immutability, and that the rule

applicator was actually creating a mutable list

4. realized component metadata wasn't setting the `force` flag properly for

enforced platforms

This fixes #6978

    • -6
    • +2
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 10 more files in changeset.
Avoid the creation of a string to build the path in the file store

This creates a significant amount of garbage. Assuming we read from the store

more often than we write to, this commit builds an array corresponding to the

path in the store, rather than a string.

  1. … 3 more files in changeset.
Omit excludes for Maven optional dependencies

For BOM import support, when a <dependencyManagement> entry has a declared <exclusion>,

we apply this exclusion globally whenever the target module is resolved, regardless of

the path to the module. However, this behaviour is not desired for <optional> dependencies,

and these could not be distinguished from <dependencyManagement> entries after the POM was parsed.

With this change, we retain a 'type' attribute for all dependencies parsed from a Maven POM

file. This allows us to omit any exclusions that are defined for a dependency declared with

<optional>, while retaining them for <dependencyManagement> entries.

  1. … 8 more files in changeset.
Document lazy vs. realised ModuleComponentResolveMetadata

Also includes minor code refactoring to improve readability.

    • -0
    • +6
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 9 more files in changeset.
Rename method to better indicate meaning

The hash value returned by getOriginalContentHash() is the one from the

metadata parsed from a repository. It does not reflect any of the

potential mutations that happened to it at later stages.

The new name indicates this better.

  1. … 9 more files in changeset.
Improved name and documentation of attribute container serializer

    • -2
    • +2
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 10 more files in changeset.
Refactor ModuleResolveMetadata

Reorganize code in packages, move serialization related code closer to

classes to reduce need of public methods.

    • -462
    • +21
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 78 more files in changeset.
Refactor ModuleResolveMetadata

Reorganize code in packages, move serialization related code closer to

classes to reduce need of public methods.

    • -462
    • +21
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 78 more files in changeset.
Serialization of realised ModuleComponentResolveMetadata

This enables proper caching by making sure we can save and reload cache

values.

Fixes #5653

    • -4
    • +467
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 13 more files in changeset.
Serialization of realised ModuleComponentResolveMetadata

This enables proper caching by making sure we can save and reload cache

values.

Fixes #5653

    • -4
    • +467
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 13 more files in changeset.
Rename to ArtifactCacheLockingManager

The name CacheLockingManager suggested it was used to manage locking of

caches in general instead of just for the artifacts cache.

Resolves gradle/gradle-private#1338.

    • -3
    • +3
    ./artifacts/DefaultModuleArtifactCache.java
    • -5
    • +5
    ./artifacts/DefaultModuleArtifactsCache.java
    • -5
    • +5
    ./dynamicversions/DefaultModuleVersionsCache.java
  1. … 25 more files in changeset.
Realised metadata

This includes the following:

* Realising variants, by applying attributes, capabilities and

dependency rules

* Eagerly creating configurations and applying attributes, capabilites

and dependency rules

Issue #5653

    • -1
    • +21
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 17 more files in changeset.
Realised metadata

This includes the following:

* Realising variants, by applying attributes, capabilities and

dependency rules

* Eagerly creating configurations and applying attributes, capabilites

and dependency rules

Issue #5653

    • -1
    • +21
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 20 more files in changeset.
Normalize `ModuleIdentifier`

This commit reworks the `ComponentModuleIdentifier`/`ComponentModuleSelector`/`ModuleVersionSelector`

classes to use `ModuleIdentifier` under the hood, instead of storing denormalized strings. This has

the advantage that we can reduce the use of the module identifier factory, which is called very

often during dependency resolution. Sharing instances reduces the need for conversions, and makes

comparisons faster.

  1. … 164 more files in changeset.
Track file access in AbstractCachedIndex

Instead of tracking access in DefaultCacheAwareExternalResourceAccessor

and CachingModuleComponentRepository this is now done in

AbstractCachedIndex which ensures it's done always when reading from

indexes of CachedItems and only does so when not already present in an

in-memory cache from a previous lookup.

    • -2
    • +3
    ./artifacts/DefaultModuleArtifactCache.java
  1. … 38 more files in changeset.
Resurrect and fix DefaultModuleArtifactCacheTest

    • -0
    • +2
    ./artifacts/DefaultModuleArtifactCache.java
  1. … 2 more files in changeset.
Avoid resizing of linked hash set when we know the number of elements

Don't clean up metadata descriptors

Metadata descriptors are small (about 4 KB) and thus don't need to be

cleaned up.

  1. … 3 more files in changeset.
Track artifact cache file access in PersistentIndexedCache

This commit introduces the `FileAccessJournal` interface and provides

two implementations:

ModificationTimeFileAccessJournal::

Reads and sets `File.lastModified()`.

IndexedCacheBackedFileAccessJournal::

Uses a PersistentIndexedCache to store the access timestamp.

The latter is now used in DefaultCacheLockingManager for the artifact

cache. All other PersistentCaches still use the former.

  1. … 29 more files in changeset.
Mark cached files accessed when resolved via index

Both CachingModuleComponentRepository and

DefaultCacheAwareExternalResourceAccessor use an index to resolve

cached artifacts. Both now notify the underlying FileStore which now

implements FileAccessTracker. The new AccessTrackingFileStore interface

combines both interfaces so the GroupedAndNamedUniqueFileStore

subclasses (ArtifactIdentifierFileStore and ExternalResourceFileStore)

can be injected into dependent components.

  1. … 32 more files in changeset.
Clean up least recently used entries in module metadata store

The files in the module metadata store cache are now automatically

cleaned up based on a least recently used strategy. The structure in

the cache is as follows:

- modules-2

- metadata-2.58

- descriptors

- groupId

- artifactId

- version

- hash1 (*)

- file

- hash2 (*)

- file

The cache now keeps track of files accessed inside the marked hash

directories. The cleanup action then deletes all such files that have

not been accessed in the last 30 days.

  1. … 2 more files in changeset.
Create and wire in a ComponentMetadataRuleExecutor

This enables ComponentMetadataRule execution to be cached

Update the CrossBuildCachingRuleExecutor to work with a set of rule

instead of a single one. This means we cache or miss on a chain of rules

instead of single rules.

Fixes #5526

    • -0
    • +47
    ./ModuleComponentResolveMetadataSerializer.java
  1. … 32 more files in changeset.
Add initial support component metadata supplier caching

This commit adds the infrastructure for caching of component metadata rules.

For now, the only kind of rule we can cache are the component metadata supplier

rules, which are used during dynamic version selection, but the executor

infrastructure can be reused to cache more kinds of rules.

The implementation relies on the principal that a "rule" can be safely cached if:

- the rule has no side effect, and uses the "class-based pattern"

- the implementation of the rule didn't change

- the inputs of the rule didn't change

For this, we introduce a new persistent cache (hence cross-build) with an in-memory

facing cache, that allows us to avoid the execution of a rule if the result is

already in the cache. The implementation makes use of _snapshotting_ to capture the

state of the inputs of a rule, which consists of the implementation of the rule and

its potential parameters. It's worth noting that at this stage we do not consider

the services the rule can use, it's going to be done in a subsequent commit.

This worked required a clarification of what an rule cares about (the result) in

opposition to what the user is faced with. For example, a component metadata supplier

rule output is a `ComponentMetadata`, but what we offer to the user, to build that

metadata, is a `ComponentMetadataSupplierDetails` instance. Similarly, component

metadata rules would have different kind of output and what the user manipulates.

The cache works with a primary key (for the first implemented cache, the key is the

module version identifier) and will invalidate the results based on the cache policy

(should refresh modules).

The persistent cache uses the snapshot as the key in the store. In theory, should we

consider that we have a nearly perfect hash function, we could instead use the hash

of the snapshot as the key. Measurements would have to be made to check if it's worth

implementing.

    • -0
    • +87
    ./SuppliedComponentMetadataSerializer.java
  1. … 46 more files in changeset.
Add initial support component metadata supplier caching

This commit adds the infrastructure for caching of component metadata rules.

For now, the only kind of rule we can cache are the component metadata supplier

rules, which are used during dynamic version selection, but the executor

infrastructure can be reused to cache more kinds of rules.

The implementation relies on the principal that a "rule" can be safely cached if:

- the rule has no side effect, and uses the "class-based pattern"

- the implementation of the rule didn't change

- the inputs of the rule didn't change

For this, we introduce a new persistent cache (hence cross-build) with an in-memory

facing cache, that allows us to avoid the execution of a rule if the result is

already in the cache. The implementation makes use of _snapshotting_ to capture the

state of the inputs of a rule, which consists of the implementation of the rule and

its potential parameters. It's worth noting that at this stage we do not consider

the services the rule can use, it's going to be done in a subsequent commit.

This worked required a clarification of what an rule cares about (the result) in

opposition to what the user is faced with. For example, a component metadata supplier

rule output is a `ComponentMetadata`, but what we offer to the user, to build that

metadata, is a `ComponentMetadataSupplierDetails` instance. Similarly, component

metadata rules would have different kind of output and what the user manipulates.

The cache works with a primary key (for the first implemented cache, the key is the

module version identifier) and will invalidate the results based on the cache policy

(should refresh modules).

The persistent cache uses the snapshot as the key in the store. In theory, should we

consider that we have a nearly perfect hash function, we could instead use the hash

of the snapshot as the key. Measurements would have to be made to check if it's worth

implementing.

    • -0
    • +87
    ./SuppliedComponentMetadataSerializer.java
  1. … 46 more files in changeset.
Intern strings when reading module metadata from cache

Dependency resolution of large dependency graphs involves a significant

number of comparisons of strings(group, artifact, version, ...). Most of

those come from the module metadata cache, and even if we use hashmaps,

we still need to perform `equals` comparisons on strings, when in most

of the cases they should be identical. This commit takes advantage of

knowing that to add a cost when we read module metadata (interning), but

realizing that the debt is paid when comparing strings during resolution.

The interner is build scoped (in order to avoid memory leaks), thread-safe,

and shared with the dependency notation converter, so that module selectors

created from strings found in the build scripts are using the same strings

as the ones from the module metadata cache.

Ideally, we should also do this for the strings used during parsing.

    • -0
    • +110
    ./StringDeduplicatingDecoder.java
  1. … 15 more files in changeset.
Serialize dependency attributes metadata

Now that module metadata may contain dependencies with attributes, we need to serialize them too.

This commit updates the component selector serializer, as well as the module metadata serializer,

to use this information, effectively bumping the cache metadata layout format version.

  1. … 18 more files in changeset.
Read dependency attributes in Gradle module metadata

This commit adds support for parsing dependency attributes in Gradle module metadata.

It is not yet making use of that information, it's only making sure that the parser

is able to parse and propagate it. It doesn't touch the serializer either, which is

going to be done in a separate commit.

Module metadata file version has been bumped to 0.4 as part of this work.

  1. … 17 more files in changeset.