TestNGExecutionResult.groovy

Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Ensure that all children TestState are completed() (#4752)

* Ensure that all children TestState are completed()

With fail fast for tests enabled, some of the `TestState`s tracked

by the `StateTrackingTestResultProcessor` may never receive the

`completed()` call. When this happens, the parent/ancester

`TestState` will not include the test count and failure information.

When combined with a `--tests` filter, this can cause an erroneous

failure message that no tests matched the filter.

This change ensures that all of the children/descendents of a `testId`

(`TestState`) are `completed` before the given `testId` is processed.

Addresses: #4718

* Use ArrayList<> instead of LinkedList<>

  1. … 7 more files in changeset.
Add displayName and classDisplayName to TestDescriptor (#4425)

Add displayName and classDisplayName to TestDescriptor

This fixes https://github.com/gradle/gradle/issues/4424 and https://github.com/gradle/gradle/issues/4423

JUnit 5 introduces @DisplayName and dynamic tests, which allows users to

customize test case and test class' name. This should be taken into

account. This PR introduces `displayName` and `classDisplayName` which are

used for display. When rendering HTML reports, these two fields will be used.

  1. … 33 more files in changeset.
Stop test execution after first failure (#4190)

* Rough pass at stopNow() on test failure

* Updated worker TestClassProcessor.stopNow() to throw UnsupportedOperationException

* Updated MaxNParallelTestClassProcessor to keep "raw" processors for stopNow().

Updated other daemon-side TestClassProcessors to keep track if stopNow() is called.

* Added AbstractTestTask.getFailFast()

* Added some unit tests for to TestClassProcessors

* Added unit tests for stopNow()

* Rough pass at JUnitFailFastIntegrationTest

* Refactor fail fast JVM integration test.

Moved common logic/tests from JUnitFailFastIntegrationTest into AbstractJvmFailFastIntegrationSpec.

Added TestNGFailFastIntegrationTest (which extends TestNGFailFastIntegrationTest)

* Working on forkEvery fail fast test

* Converted FailFastTestListener to FailFastTestListenerInternal

* Added BlockingHttpServer.expectMaybeAndBlock()

* TestNG parallel & fail-fast integration test

* Remove unused import

* Remove unused import

* Marked test.failFast as @Input

* Added `failFast` to java_plugin in userguide

* Javadoc & formatting changes

* Updated userguide docs based on review comments

* Moved `failFast` configuration from `AbstractTestTask` to `Test` to avoid `XCTest`

* Updates from review comments

* More updates from review comments

* Reduced Mock() usage in FailFastTestListenerInternalTest

* Backed out changes to XCTestExecuter for fail fast behavior

* Fixed typo in javadoc

* Remove --no-fail-fast `@Option` from `Test`

* Reduce mocking expectations in ForkingTestClassProcessorTest

* Moved @Internal from Test.getFailFast() to AbstractTestTask

* Formatting updates

* Updated ForkingTestClassProcessor to track state more precisely to avoid stop() & stopNow() problem

* Fixed tests (again)

* Better handling of mutual exclusion between ForkingTestClassProcessor stop() and stopNow() sections

* Improved resiliency to indeterminate generated test class execution in fail fast tests

* Optimized imports

* Updated DefaultWorkerProcess.cleanup() to stop the Stoppables before aborting the execHandle

* Changed CyclicBarrierAnyOfRequestHandler.expected back to `private`

* ForkingTestClassProcessor.stoppedNow does not need to be `volatile` as it is now protected by a lock

* Added JUnitPlatformTestClassProcessor.stopNow()

* Removed mention of --no-fail-fast from user guide

* Added info on --fail-fast to release notes

* Fixed use of `testOmitted` in AbstractJvmFailFastIntegrationSpec

* Make MaxNParallelTestClassProcessor drop any processTestClass() invocations after stopNow()

* Protected critical region in ForkingTestClassProcessor.processTestClass() causing race condition with stopNow() in tests

* Debugging cleanup

  1. … 41 more files in changeset.
Add more test coverage for XcTest and testing frameworks

- When a test class is removed, make sure the intermediate report files are also removed.

- When no tests are found, make sure the test results have no classes listed.

- When tests are executed, make sure that only those tests are included in the report.

- When a test fails, make sure the failure is included in the report.

  1. … 11 more files in changeset.
Fix some tests that rely on a test executor with a specific ID

  1. … 14 more files in changeset.
include task name in test report/result folder when using java plugin

  1. … 11 more files in changeset.
Fail test execution when non-default config failure policy is used with a TestNG version that doesn't support it

+review REVIEW-5274

  1. … 8 more files in changeset.
Changed some source files so that they will compile under Groovy 2.x

  1. … 7 more files in changeset.
GRADLEREV-15 - add TestNGOptions.outputDirectory property.

This decouples TestNG's native output from Gradle's.

  1. … 5 more files in changeset.
Changed the HtmlTestExecutionResult assert methods so that they actually assert stuff, and some other minor fixes in the various TestExecutionResult implementations.

  1. … 3 more files in changeset.
Remove @author tags and names from source code.

- Added checkstyle check for @author

- Added not to CONTRIBUTING.md saying that we don't use names in the codebase

  1. … 1213 more files in changeset.
Enabled generating the JUnit XML file with output per test case.

This is not complete. The actual generation of the XML based on the TestResultsProvider is unit tested and there is functional level coverage when running TestNG tests.

Coverage for JUnit is needed, as well as coverage at the intermediary parts of the results/report generation.

There are also some types to tidy up.

  1. … 17 more files in changeset.
Some renaming and cleanup around the test/temp directory used in tests.

  1. … 308 more files in changeset.
TestNG reports are now lovely (not by default, though)

1. By default the TestNG works as it was before, e.g. all 3 old reports are generated into usal spots. Those reports are generated by *TestNG*. Those reports do not contain test output and fail when forkEvery or maxParallelForks is used.

2. Only if the user configures the test.testReport = true in his buildscript the new reports are generated. If testReport is on, the old TestNG reports are *not* generated. It is somewhat a breaking change so it needs proper documentation and announcement. The new reports are generated by *Gradle* which means they are beautiful, they contain test output and work smoothly with forkEvery / maxParallelForks. The new reports consists of: new xml results (with test outputs, CI servers will be happy), new html reports (pretty, easy to read and navigate).

3. If (unlikely) some user really wants to generate both reports (old and new) it is possible, he needs to configure the default listeners via TestNGOptions.listeners

4. Setting TestNGOptions.useDefultListeners = true has no effect if the testReport is on. A mildly breaking change.

5. It is possible to prevent generation of both old and new reports via TestNGOptions.useDefultListeners = false and test.testReport = false

The new reports are not yet turned on by default because they consume tons of heap space. It's because with TestNG we don't receive events for start/end test class and hence we need to wait until the suite end to generate the xml results. This problem can be fixed, for example by making the serialization smarter. However, it's not yet implemented.

The proposed breaking changes are mild IMHO. The default behavior stays untouched. Also, the migration plan to use the new reports by default should be fairly straightforward. The changes are more-less what we've discussed earlier with Adam; also discussed with PeterN today. Please review and thanks for reading :)

  1. … 10 more files in changeset.
Improving TestNG reports. First, some refactoring.

1. Extracted the JUnit xml report serialization from the JUnitXmlReportGenerator so that I can reuse it for TestNG. I'd rather avoid reusing parts of StateTrackingTestResultProcessor at this time.

2. Tightened the integration tests a little bit. Some new methods in the fixture code are not yet used but they will be needed soon (it was easier to include it now).

  1. … 5 more files in changeset.
Fixed the fully qualified class name after the refactoring.

Refactoring and restructruization. Moved the samples integ tests into a spearate package for better visibility and runability. Moved test fixture code into the integ test internal subproject.

    • -0
    • +143
    ./TestNGExecutionResult.groovy
  1. … 65 more files in changeset.