Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Introduce a checksum file cache service

This service is responsible for caching the checksums computed from

local file system. Because it's also used for dependency verification

writing and checking, this cache uses the existing infrastructure which

makes sure that if a file is updated locally, we expire the entry in

the cache.

This is done because there are lots of places in the code where we

used the legacy `HashUtil` class, which has no caching whatsoever.

It's, however, quite common to have a build which generates sha1

checksums multiple times for the same file. For example, during

publication.

    • -8
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 102 more files in changeset.
Introduce a checksum file cache service

This service is responsible for caching the checksums computed from

local file system. Because it's also used for dependency verification

writing and checking, this cache uses the existing infrastructure which

makes sure that if a file is updated locally, we expire the entry in

the cache.

This is done because there are lots of places in the code where we

used the legacy `HashUtil` class, which has no caching whatsoever.

It's, however, quite common to have a build which generates sha1

checksums multiple times for the same file. For example, during

publication.

    • -8
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 102 more files in changeset.
Introduce a checksum file cache service

This service is responsible for caching the checksums computed from

local file system. Because it's also used for dependency verification

writing and checking, this cache uses the existing infrastructure which

makes sure that if a file is updated locally, we expire the entry in

the cache.

This is done because there are lots of places in the code where we

used the legacy `HashUtil` class, which has no caching whatsoever.

It's, however, quite common to have a build which generates sha1

checksums multiple times for the same file. For example, during

publication.

    • -8
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 102 more files in changeset.
Introduce a checksum file cache service

This service is responsible for caching the checksums computed from

local file system. Because it's also used for dependency verification

writing and checking, this cache uses the existing infrastructure which

makes sure that if a file is updated locally, we expire the entry in

the cache.

This is done because there are lots of places in the code where we

used the legacy `HashUtil` class, which has no caching whatsoever.

It's, however, quite common to have a build which generates sha1

checksums multiple times for the same file. For example, during

publication.

    • -8
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 103 more files in changeset.
Serialize module sources when realizing component metadata

    • -1
    • +3
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 7 more files in changeset.
Serialize module sources when realizing component metadata

    • -1
    • +3
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 6 more files in changeset.
Always serialize module sources

Module sources were only serialized in the cache metadata entry.

In practice, they belong to the module metadata, so they are now

properly serialized as part of it. This fixes the "force realize"

tests.

    • -0
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 18 more files in changeset.
Always serialize module sources

Module sources were only serialized in the cache metadata entry.

In practice, they belong to the module metadata, so they are now

properly serialized as part of it. This fixes the "force realize"

tests.

    • -0
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 18 more files in changeset.
Always serialize module sources

Module sources were only serialized in the cache metadata entry.

In practice, they belong to the module metadata, so they are now

properly serialized as part of it. This fixes the "force realize"

tests.

    • -0
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 18 more files in changeset.
Introduce module metadata verification

This commit introduces verification of metadata files.

For this, another refactoring of the `ModuleSource` concept

was required. Ironically, before this commit, `ModuleSource`

used to be available for serialization in the module metadata

serializer. However, they weren't used in practice, because

all the required data could be reconstructed from the caches.

In particular, there was this "contentHash" which, because

not properly serialized, was actually set as a field on the

component resolve metadata itself, instead of being part

of the module source.

Now, this commit reintroduces serialization of module sources

but takes a different approach by splitting the module sources

in two distinct categories:

- module sources which can be reconstructed from known data,

such as the repository name and repository url

- module sources which have to be serialized alongside component

metadata, because they can't be reconstructed from sources

The latter category includes this "contentHash", serialized with

the descriptor hash module source. It also includes the information

about _which_ actual descriptor file was used to generate the

binary module descriptor (e.g, the source POM, Ivy or module

metadata file). This information does _not_ belong to the module

component resolve metadata itself, so it belongs to its sources.

For this purpose, serialization of module sources has been updated

so that instead of using Java serialization, module sources need

to provide a custom serializer, called a Codec. Those codecs are

uniquely identified by an id which is just an integer. This is

done for performance optimization, in order to avoid to serialize

the name of the codec and have to load it dynamically. Instead,

Gradle knows about the full set of serializers (and there's no

way for a user to add more because in any case it would require

an update of the module metadata store format).

This makes it much more efficient to serialize module sources

(because we can now have an optimized encoder), but it also

permits reconstructing module sources from incomplete information.

In particular, the module source which describes the file from

which a component resolve metadata was sourced contains a link

to the actual file in the local artifact store. However, in order

to be relocatable, we _don't_ want this file path to be stored

in the metadata cache. This means that instead of storing the

path, we actually store the artifact identifier and the hash

of the descriptor so that we can, when loaded from cache, find

its location back.

Currently, metadata verification is enabled for all components.

It's not possible to disable verification of metadata.

    • -3
    • +9
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 39 more files in changeset.
Introduce module metadata verification

This commit introduces verification of metadata files.

For this, another refactoring of the `ModuleSource` concept

was required. Ironically, before this commit, `ModuleSource`

used to be available for serialization in the module metadata

serializer. However, they weren't used in practice, because

all the required data could be reconstructed from the caches.

In particular, there was this "contentHash" which, because

not properly serialized, was actually set as a field on the

component resolve metadata itself, instead of being part

of the module source.

Now, this commit reintroduces serialization of module sources

but takes a different approach by splitting the module sources

in two distinct categories:

- module sources which can be reconstructed from known data,

such as the repository name and repository url

- module sources which have to be serialized alongside component

metadata, because they can't be reconstructed from sources

The latter category includes this "contentHash", serialized with

the descriptor hash module source. It also includes the information

about _which_ actual descriptor file was used to generate the

binary module descriptor (e.g, the source POM, Ivy or module

metadata file). This information does _not_ belong to the module

component resolve metadata itself, so it belongs to its sources.

For this purpose, serialization of module sources has been updated

so that instead of using Java serialization, module sources need

to provide a custom serializer, called a Codec. Those codecs are

uniquely identified by an id which is just an integer. This is

done for performance optimization, in order to avoid to serialize

the name of the codec and have to load it dynamically. Instead,

Gradle knows about the full set of serializers (and there's no

way for a user to add more because in any case it would require

an update of the module metadata store format).

This makes it much more efficient to serialize module sources

(because we can now have an optimized encoder), but it also

permits reconstructing module sources from incomplete information.

In particular, the module source which describes the file from

which a component resolve metadata was sourced contains a link

to the actual file in the local artifact store. However, in order

to be relocatable, we _don't_ want this file path to be stored

in the metadata cache. This means that instead of storing the

path, we actually store the artifact identifier and the hash

of the descriptor so that we can, when loaded from cache, find

its location back.

Currently, metadata verification is enabled for all components.

It's not possible to disable verification of metadata.

    • -3
    • +9
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 38 more files in changeset.
Introduce module metadata verification

This commit introduces verification of metadata files.

For this, another refactoring of the `ModuleSource` concept

was required. Ironically, before this commit, `ModuleSource`

used to be available for serialization in the module metadata

serializer. However, they weren't used in practice, because

all the required data could be reconstructed from the caches.

In particular, there was this "contentHash" which, because

not properly serialized, was actually set as a field on the

component resolve metadata itself, instead of being part

of the module source.

Now, this commit reintroduces serialization of module sources

but takes a different approach by splitting the module sources

in two distinct categories:

- module sources which can be reconstructed from known data,

such as the repository name and repository url

- module sources which have to be serialized alongside component

metadata, because they can't be reconstructed from sources

The latter category includes this "contentHash", serialized with

the descriptor hash module source. It also includes the information

about _which_ actual descriptor file was used to generate the

binary module descriptor (e.g, the source POM, Ivy or module

metadata file). This information does _not_ belong to the module

component resolve metadata itself, so it belongs to its sources.

For this purpose, serialization of module sources has been updated

so that instead of using Java serialization, module sources need

to provide a custom serializer, called a Codec. Those codecs are

uniquely identified by an id which is just an integer. This is

done for performance optimization, in order to avoid to serialize

the name of the codec and have to load it dynamically. Instead,

Gradle knows about the full set of serializers (and there's no

way for a user to add more because in any case it would require

an update of the module metadata store format).

This makes it much more efficient to serialize module sources

(because we can now have an optimized encoder), but it also

permits reconstructing module sources from incomplete information.

In particular, the module source which describes the file from

which a component resolve metadata was sourced contains a link

to the actual file in the local artifact store. However, in order

to be relocatable, we _don't_ want this file path to be stored

in the metadata cache. This means that instead of storing the

path, we actually store the artifact identifier and the hash

of the descriptor so that we can, when loaded from cache, find

its location back.

Currently, metadata verification is enabled for all components.

It's not possible to disable verification of metadata.

    • -3
    • +9
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 39 more files in changeset.
Introduce module metadata verification

This commit introduces verification of metadata files.

For this, another refactoring of the `ModuleSource` concept

was required. Ironically, before this commit, `ModuleSource`

used to be available for serialization in the module metadata

serializer. However, they weren't used in practice, because

all the required data could be reconstructed from the caches.

In particular, there was this "contentHash" which, because

not properly serialized, was actually set as a field on the

component resolve metadata itself, instead of being part

of the module source.

Now, this commit reintroduces serialization of module sources

but takes a different approach by splitting the module sources

in two distinct categories:

- module sources which can be reconstructed from known data,

such as the repository name and repository url

- module sources which have to be serialized alongside component

metadata, because they can't be reconstructed from sources

The latter category includes this "contentHash", serialized with

the descriptor hash module source. It also includes the information

about _which_ actual descriptor file was used to generate the

binary module descriptor (e.g, the source POM, Ivy or module

metadata file). This information does _not_ belong to the module

component resolve metadata itself, so it belongs to its sources.

For this purpose, serialization of module sources has been updated

so that instead of using Java serialization, module sources need

to provide a custom serializer, called a Codec. Those codecs are

uniquely identified by an id which is just an integer. This is

done for performance optimization, in order to avoid to serialize

the name of the codec and have to load it dynamically. Instead,

Gradle knows about the full set of serializers (and there's no

way for a user to add more because in any case it would require

an update of the module metadata store format).

This makes it much more efficient to serialize module sources

(because we can now have an optimized encoder), but it also

permits reconstructing module sources from incomplete information.

In particular, the module source which describes the file from

which a component resolve metadata was sourced contains a link

to the actual file in the local artifact store. However, in order

to be relocatable, we _don't_ want this file path to be stored

in the metadata cache. This means that instead of storing the

path, we actually store the artifact identifier and the hash

of the descriptor so that we can, when loaded from cache, find

its location back.

Currently, metadata verification is enabled for all components.

It's not possible to disable verification of metadata.

    • -3
    • +9
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 39 more files in changeset.
Allow the services required by a given class to be queried prior to creating any instances of that class. Use this to allow `ArtifactTransformDependencies` to be injected into artifact transforms using any of the service injection patterns (that is, via a constructor or a getter).

    • -2
    • +2
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 127 more files in changeset.
Move in-memory caches to persistent-cache project

    • -1
    • +1
    ./ComponentMetadataRuleExecutorTest.groovy
    • -1
    • +1
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -1
    • +1
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 48 more files in changeset.
Move value snapshot implementations to snapshots project

    • -2
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
    • -1
    • +1
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -1
    • +1
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 94 more files in changeset.
Move ValueSnapshotter and friends to snapshots project

    • -2
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
    • -2
    • +2
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -2
    • +2
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 52 more files in changeset.
Remove unused improve pom support flags

    • -1
    • +1
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 19 more files in changeset.
Use MD5 as the default hashing function

    • -1
    • +1
    ./ComponentMetadataRuleExecutorTest.groovy
    • -1
    • +1
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -2
    • +2
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 36 more files in changeset.
Move all hashing-related stuff to base-services

- Renamed existing Hasher -> PrimitiveHasher

- Renamed BuildCacheHasher -> Hasher (this is the one that prefixes hashed data with the length of the data to avoid collisions)

    • -2
    • +2
    ./ComponentMetadataRuleExecutorTest.groovy
    • -2
    • +2
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -3
    • +3
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 70 more files in changeset.
Add pom support feature to cache key for metadata rules

The IMPROVED_POM_SUPPORT feature preview has a direct impact on metadata

resolution. As such, changing that value must invalidate the component

metadata rule cache.

    • -3
    • +3
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 5 more files in changeset.
Rename method to better indicate meaning

The hash value returned by getOriginalContentHash() is the one from the

metadata parsed from a repository. It does not reflect any of the

potential mutations that happened to it at later stages.

The new name indicates this better.

    • -1
    • +1
    ./ComponentMetadataRuleExecutorTest.groovy
  1. … 9 more files in changeset.
Update CrossBuildCachingRuleExecutor key value

It now uses the hash of the ValueSnapshot instead of the whole object.

This allows rules taking an Attribute parameter to be cached.

However it also adds a test showing that we do not yet support caching

of rules transforming variant attributes.

    • -2
    • +6
    ./ComponentMetadataRuleExecutorTest.groovy
    • -2
    • +6
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -2
    • +10
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 3 more files in changeset.
Update CrossBuildCachingRuleExecutor key value

It now uses the hash of the ValueSnapshot instead of the whole object.

This allows rules taking an Attribute parameter to be cached.

However it also adds a test showing that we do not yet support caching

of rules transforming variant attributes.

    • -2
    • +6
    ./ComponentMetadataRuleExecutorTest.groovy
    • -2
    • +6
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -2
    • +10
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 3 more files in changeset.
Make ConfigurableRule expose isolated rule params

This will better integrate with caching as the isolation will be

performed once instead of multiple times for a single rule.

Fixes #5261

    • -2
    • +1
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -3
    • +3
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 15 more files in changeset.
Create and wire in a ComponentMetadataRuleExecutor

This enables ComponentMetadataRule execution to be cached

Update the CrossBuildCachingRuleExecutor to work with a set of rule

instead of a single one. This means we cache or miss on a chain of rules

instead of single rules.

Fixes #5526

    • -0
    • +236
    ./ComponentMetadataRuleExecutorTest.groovy
    • -7
    • +2
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -6
    • +54
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 30 more files in changeset.
Implement discovery of rule inputs for injected services

This commit replaces the not-so-nice service "snapshotting" with a proper

implementation of input discovery for services. The idea is that rules can

have injected services. Those services are usually blackboxes, but they

may actually do work which is "expirable". Typically, a rule which fetches

an external resource will need to re-execute once the resource is outdated.

The problem is that the service is called _at execution time_, so we only

know about the actual rule inputs once it has effectively been executed.

To implement this capture, we need services to declare that if called,

they will participate in some inputs. In that case, we record a "query"

to the service, as well as the fingerprint of the answer to the query. Then

the result of the rule is cached. Once we fetch the result from the cache,

we ask the service if the result for the cached query is up-to-date. If not,

we discard the cache entry, otherwise, we consider it's up-to-date.

This removes the need for "snapshotting services" as we consider that a

service which doesn't declare itself as contributing inputs has no side

effect (it will always return the same value, independently of time).

    • -41
    • +53
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
    • -2
    • +76
    ./CrossBuildCachingRuleExecutorTest.groovy
  1. … 12 more files in changeset.
Implement a 24 hours timeout for rule caching

This is a (hopefully) temporary workaround for not being able to detect

implicit inputs, the ones that are provided by services, and that may

expire after a different timeout than the module resolution lifetime.

    • -18
    • +36
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
  1. … 1 more file in changeset.
Implement a 24 hours timeout for rule caching

This is a (hopefully) temporary workaround for not being able to detect

implicit inputs, the ones that are provided by services, and that may

expire after a different timeout than the module resolution lifetime.

    • -18
    • +36
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
  1. … 1 more file in changeset.
Introduce `ComponentMetadataSupplierRuleExecutor`

This gives us better abilities to unit test the behavior of this specific

rule executor.

    • -0
    • +202
    ./ComponentMetadataSupplierRuleExecutorTest.groovy
  1. … 16 more files in changeset.