Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Make file lock packet type explicit

`FileLockCommunicator` now includes an additional byte that represents

the `FileLockPacketType` when sending packets to other processes.

`DefaultFileLockContentionHandler` now checks for the

`LOCK_RELEASE_CONFIRMATION` type instead of interpreting any packet for

a lock for which it already has received a packet as a lock release

confirmation.

Adding the additional byte will be ignored by old versions which only

read the first 9 bytes. New versions can still read packets from old

versions that do not include the byte. They use the `UNKNOWN` type in

this case. Thus, this protocol change is backwards compatible.

  1. … 4 more files in changeset.
Ensure file lock is released before cache cleanup

Caches with on-demand locking keep hold of file locks unless contended

by other Gradle processes. Prior to this commit, such previously

acquired file locks were held for the complete duration of cache

cleanup only to be released immediately afterwards. While cache cleanup

was running, requests to release such file locks from other processes

(via `DefaultFileLockContentionHandler`) were not processed because the

`stateLock` of `DefaultCacheAccess` was already locked by another

thread and thus, the `ContentionAction` in

`LockOnDemandCrossProcessCacheAccess` was blocked until the `close()`

method in `DefaultCacheAccess` was finished.

Since cache cleanup only deletes files that are no longer in use, it

does not use file locking (which would not work anyway because the

current cache access infrastructure is not usable while the cache is

already being closed). Thus, this commit closes the

`CrossProcessCacheAccess` of the `DefaultCacheAccess` *before* cleaning

up caches.

Issue: gradle/gradle-private#1412

  1. … 1 more file in changeset.
Improve file lock contention handling

This commit improves the likelihood for lock requesters to acquire a

file lock after it is released due to contention. After the lock has

been released, the former lock holder now sends a packet to the sockets

of all requesters. While old clients will simply ignore the additional

packet, new clients will interpret it as a signal that the file lock has

been released and will try to acquire it immediately.

Issue: gradle/gradle-private#1412.

    • -0
    • +74
    ./gradle/cache/internal/DefaultFileLockManagerAwaitableFileLockReleasedSignalTest.groovy
    • -35
    • +56
    ./gradle/cache/internal/LockOnDemandCrossProcessCacheAccessTest.groovy
  1. … 10 more files in changeset.
Report deleted and skipped cache entries

This commit introduces `CleanupProgressMonitor` and a default

`ProgressLogger`-based implementation that tracks the total number of

deleted and skipped cache entries. `CleanupAction.clean()` now takes a

monitor as an additional argument. `DefaultPersistentDirectoryStore`

creates the default implementation and passes it to its `CleanupAction`.

    • -0
    • +58
    ./gradle/cache/internal/DefaultCleanupProgressMonitorTest.groovy
  1. … 10 more files in changeset.
Use ProgressLogger so cache cleanup is visible

    • -12
    • +14
    ./gradle/cache/internal/DefaultPersistentDirectoryCacheTest.groovy
    • -1
    • +2
    ./gradle/cache/internal/DefaultPersistentDirectoryStoreConcurrencyTest.groovy
  1. … 7 more files in changeset.
Make cache cleanup non-incremental

  1. … 10 more files in changeset.
Avoid reinitializing DefaultPersistentDirectoryStores

Prior to this commit, adding a `CleanupAction` when building a

`PersistentCache` using `CacheBuilder.withCleanup()` caused the used

implementation class to be changed from

`DefaultPersistentDirectoryStore` to `DefaultPersistentDirectoryCache`.

The latter adds initialization logic and has a very strict check in

place that verifies whether the lock file has been unlocked cleanly. If

not, it will delete all files in the cache in order to reinitialize it.

Since the mere addition of a cleanup action should not change such a

fundamental behavior, this commit moves the cleanup logic to the

`DefaultPersistentDirectoryStore` class and instantiates it when a

cleanup action is configured but no initialization related properties.

    • -1
    • +1
    ./gradle/cache/internal/DefaultPersistentDirectoryStoreConcurrencyTest.groovy
  1. … 4 more files in changeset.
Rename GradleVersionProvider to UsedGradleVersions

  1. … 11 more files in changeset.
Handle multi-component cache versions

- Let `UnusedVersionsCacheCleanup` handle cache versions with multiple

components, e.g. metadata-2.58.

- Introduce `CacheVersion` to handle formatting and parsing in one

place.

    • -0
    • +84
    ./gradle/cache/internal/CacheVersionTest.groovy
  1. … 10 more files in changeset.
Stop using atomic types in non-thread-safe contexts (#5813)

- Replace AtomicReference with MutableReference

- Replace AtomicBoolean with MutableBoolean

  1. … 17 more files in changeset.
Implement cleanup of shared versioned caches

`UnusedVersionsCacheCleanup` now extends `AbstractCacheCleanup` to reuse

its timeout checking and to make it usable as a `CleanupAction` for a

`PersistentCache`. `CompositeCleanupAction` now allows to use multiple

top-level `CleanupActions`.

    • -0
    • +122
    ./gradle/cache/internal/CacheVersionMappingTest.groovy
    • -0
    • +85
    ./gradle/cache/internal/UnusedVersionsCacheCleanupTest.groovy
  1. … 7 more files in changeset.
Delete empty parent dirs of deleted cache entries

Prior to this commit parent directories of deleted cache entries were

not checked whether they're eligible for deletion. Now, parent

directories that are now empty are checked up to the base directory.

For the artifact transforms cache the following directories marked with

(+) are now also deleted once all subdirectories (*) have been deleted:

- files-1.1

- artifact-a (+)

- hash 1 (*)

- hash 1.1

- ...

- hash 2 (*)

- ...

- ... (+)

For the artifact cache:

- modules-2

- files-2.1

- groupId (+)

- artifactId (+)

- version (+)

- hash1 (*)

- file

- hash2 (*)

- file

- resources-2.1

- 0 (+)

- hash 1 (*)

- hash 2 (*)

- ...

- ... (+)

Resolves gradle/gradle-private#1339.

  1. … 3 more files in changeset.
Improve test readability

  1. … 2 more files in changeset.
Delete access times for deleted files from journal

When a file is deleted by the LeastRecentlyUsedCacheCleanup action, the

corresponding entry is now removed from the used FileAccessTimeJournal

in order to avoid storing obsolete entries that would never get removed.

  1. … 6 more files in changeset.
Use separate cache for file access time journal

Storing a file access time journal inside each cache causes a lot of

contention in particular with caches that require a lot of exclusive

access like the artifact cache.

Instead, a new `journal-1` cache that is managed by a user-home-scoped

service is now used to keep track of file access times for all caches

that want to use it.

  1. … 23 more files in changeset.
Only query cache's reserved files once per cleanup

    • -0
    • +41
    ./gradle/cache/internal/NonReservedFileFilterTest.groovy
  1. … 3 more files in changeset.
Enable incremental cache cleanup

Instead of computing the complete list of eligible files at once,

SingleDepthFileFinder now returns an Iterable that creates an

Iterator that will incrementally walk the file tree.

In addition, the CleanupAction.clean() operation now takes a

CountdownTimer and implementations periodically check whether the timer

has expired. If so, the current cleanup will be aborted.

DefaultPersistentDirectoryCache now uses a timeout of 20 seconds which

is well below the file locking timeout of 60 seconds.

The marker file (gc.properties) will only be updated if the complete

cleanup was able to finish without expiring the timeout.

    • -0
    • +81
    ./gradle/cache/internal/CompositeCleanupActionTest.groovy
  1. … 9 more files in changeset.
Write access time asynchronously, read synchronously

File access times are now written asynchronously while the cache is

being used. When it's about to be closed, they are now read

synchronously from the cleanup action because the cache access worker

has then already been stopped.

    • -68
    • +0
    ./gradle/cache/internal/IndexedCacheBackedFileAccessJournalTest.groovy
  1. … 30 more files in changeset.
Let `CacheKeyBuilder` accept `HashCode` components

Signed-off-by: Rodrigo B. de Oliveira <rodrigo@gradle.com>

  1. … 2 more files in changeset.
Track artifact cache file access in PersistentIndexedCache

This commit introduces the `FileAccessJournal` interface and provides

two implementations:

ModificationTimeFileAccessJournal::

Reads and sets `File.lastModified()`.

IndexedCacheBackedFileAccessJournal::

Uses a PersistentIndexedCache to store the access timestamp.

The latter is now used in DefaultCacheLockingManager for the artifact

cache. All other PersistentCaches still use the former.

    • -0
    • +68
    ./gradle/cache/internal/IndexedCacheBackedFileAccessJournalTest.groovy
    • -0
    • +72
    ./gradle/cache/internal/LeastRecentlyUsedCacheCleanupTest.groovy
  1. … 28 more files in changeset.
Return Collection instead of File[]

  1. … 4 more files in changeset.
Polish and document new top-level types

    • -0
    • +60
    ./gradle/cache/internal/SingleDepthFilesFinderTest.groovy
  1. … 16 more files in changeset.
Clean up least recently used entries in artifacts file store

The files in the artifacts file store cache are now automatically

cleaned up based on a least recently used strategy. The structure in

the cache is as follows:

- modules-2

- files-2.1

- groupId

- artifactId

- version

- hash1 (*)

- file

- hash2 (*)

- file

The cache now keeps track of files accessed inside the marked hash

directories. The cleanup action then deletes all such files that have

not been accessed in the last 30 days.

    • -0
    • +60
    ./gradle/cache/internal/SingleDepthDescendantsFileFinderTest.groovy
  1. … 14 more files in changeset.
Clean up least recently used entries in external resources file store

The files in the external resources file store cache are now

automatically cleaned up based on a least recently used strategy. The

structure in the cache is as follows:

- modules-2

- resources-2.1

- 0

- hash 1 (*)

- hash 2 (*)

- ...

- ...

The cache now keeps track of files accessed inside the marked hash

directories. The cleanup action then deletes all such files that have

not been accessed in the last 30 days.

  1. … 27 more files in changeset.
Handle missing cache.properties

Prior to this commit, the check whether initialization is required

threw a `FileNotFoundException` when cache.properties did not exist.

Since the file is only created when `DefaultPersistentDirectoryCache`

is used instead of `DefaultPersistentDirectoryStore`, this can happen

when re-using an existing cache directory that used to be accessed

using the latter but from now on will be accessed using the former.

For instance, when adding a cleanup action, `DefaultCacheFactory` will

no longer return a plain store but a Ccache.

The missing properties file will now be ignored when the cache is not

using any properties. Otherwise, it will cause the cache to be

re-initialized, i.e. all contained files will be deleted.

In addition, this commit fixes a `NullPointerException` that occurred

when a property was present in the cache's new properties but missing

in the existing cache.properties file.

  1. … 1 more file in changeset.
Clean up least recently used entries in artifact transforms cache

The files in the transforms cache are now automatically cleaned up based

on a least recently used strategy. The structure in the cache is as

follows:

- files-1.1

- artifact-a

- hash 1 (*)

- hash 1.1

- ...

- hash 2 (*)

- ...

- ...

- metadata-1.1

The cache now keeps track of files accessed inside the first hash (*)

level by touching the corresponding directory. The cleanup action then

deletes all such directories that have not been accessed in the last 7

days.

Issue: #1085

    • -0
    • +44
    ./gradle/cache/internal/NonReservedCacheFileFilterTest.groovy
  1. … 7 more files in changeset.
Prevent creatig byte array copies when hashing hash codes

Should reduce garbage creation a little.

  1. … 8 more files in changeset.
Use logical equality to determine whether a cached value needs to be updated.

  1. … 2 more files in changeset.
Use age-based strategy to clean local build cache

Signed-off-by: Lóránt Pintér <lorant@gradle.com>

  1. … 12 more files in changeset.
Clean up partial files after errors and during GC

Previously '.part' files were left behind after local cache failures, because we expected they would be useful in debugging these cases. That never happened, so we are now removing them even if there's an error, and also during garbage collection.

Signed-off-by: Lóránt Pintér <lorant@gradle.com>

  1. … 5 more files in changeset.