During branching for TYPO3 v13 the GitLab workflow
cache keys has been adjusted. However, the generic
cache key has been missed. Use the correct one now.

Resolves: #103068
Releases: 12.4
Change-Id: I16a5e7e7bfbd9f1ca4c50d8057d1edfaae5b66c1
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/82831


Tested-by: Benjamin Franzke <ben@bnf.dev>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>
Tested-by: Stefan Bürk <stefan@buerk.tech>
Reviewed-by: Benjamin Franzke <ben@bnf.dev>
Tested-by: core-ci <typo3@b13.com>

This patch is paired with the setup of a new CI [1].

The patch adapts runTests.sh to allow podman as
container environment next to docker. New CI relies
on podman.

Locally, podman can be selected as container environment
using the `-b podman` flag to runTests.sh. A "recent"
podman version - at least around 4.x - should be used.
docker is kept as default for now.

The new CI setup comes with a series of improvements
to speed up things. This makes the acceptance tests
a bit more brittle again. The patch comes with some
further stabilizations, and we may add even more with
dedicated patches later.

[1] https://git.typo3.org/typo3/CI/testing-infrastructure/

Change-Id: I206c1577335bfbefb801eaababa4b713e1edda12
Resolves: #102385
Releases: main, 12.4, 11.5
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/81744


Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>

Due a regression since GitLab runner 15.1,we cannot
use the official docker dind image to raise runner
version to keep up with the TYPO3 gitlab instance.

Because of other issues, using a custom docker dind
image with changed expose port for the healthcheck
to work around that issue is needed.

See: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29130

After Gitlab has released a proper fix for this,
this should be reverted and the official docker
dind image used again.

Resolves: #100086
Releases: main, 12.1, 11.5, 10.4
Change-Id: I4b433ba7f3a42c718b4d436cf42cdf619b9094ca
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/78028


Tested-by: core-ci <typo3@b13.com>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>
Tested-by: Stefan Bürk <stefan@buerk.tech>

Yarn has been added to TYPO3 back in 2017 because
npm shrinkwrap files were not stable and package-lock.json
and `npm ci` not yet established.

By now the yarn version in TYPO3 (v1) got outdated
and an update to yarn v3 would be required sooner
or later. The update to yarn v3 brings quite some
problems for the TYPO3 codebase:

 - Yarn v3 is executed/proxied through yarn v1
   (installed on the developer machine) – therefore the
   yarn v3 executable needs to be tracked within git.
   See https://yarnpkg.com/cli/set/version#details

 - Yarn v3 pulls in @types/node (through the "got" package) for
   unknown reasons, causing clashes with TypeScript declarations
   for requirejs (window.require vs node's require())

As package-lock.json is solid thesedays, there is no real
need to depend on yarn anymore. We basically switch
to npm because that switch is way more straightforward than
migrating to yarn v3 and it removes one dependency.

Note that `npm install` properly honors `...

The issue has been resolved in sysbox, we don't
need the patch level pin anymore.

Resolves: #98015
Related: #97570
Releases: main, 11.5, 10.4
Change-Id: I1a2080ec3e8609350c9975bad8f65e69513d5f48
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/75277


Tested-by: core-ci <typo3@b13.com>
Tested-by: Stefan Bürk <stefan@buerk.tech>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

We restricted the default retry of single jobs a while
ago to do that only for acceptance test related jobs.
This does not work out well: There are simply to many
things that occassionally false-negative due to network,
timing or other issues.
The patch adds general retry:1 for all jobs for mental
sanity.

Resolves: #97597
Related: #96999
Releases: main, 11.5
Change-Id: I408b7ef1b6c6145dc1b5b12bc4550c00dfdaa930
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/74584


Tested-by: Stefan Bürk <stefan@buerk.tech>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Georg Ringer <georg.ringer@gmail.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>
Reviewed-by: Georg Ringer <georg.ringer@gmail.com>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Latest docker:20.10-dind (currently 20.10.15)
triggers an error on CI:

Error response from daemon: failed to create shim task: OCI
runtime create failed: runc create failed: unable to start
container process: unable to apply cgroup configuration:
mkdir /sys/fs/cgroup/rdma/docker: permission denied: unknown

The image is the 'sub-container' in CI on runner hosts,
that runs all the test images like the php and database
images.

This error *may* have to do with sysbox, which is a security
layer on runner hosts to separate CI jobs from each other.
We however currently don't know exactly what is going on.

For the time being, we pin the dind image to its previous
verion docker:20.10.14-dind

Change-Id: Ie59be69680e1f444c115f2249ca8709bbfdd1e3e
Releases: main, 11.5, 10.4
Resolves: #97570
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/74541


Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: Stefan Bürk <stefan@buerk.tech>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>

Acceptance tests are the most likely tests to recover
properly when automaticly retrying on test execution
failure. CGL, integration, unit and functional tests
on the other hand are more likely to fail with real
issues and retry won't fix these issues, thus wasting
CI execution time and job slots.

The patch changes the default retry back to zero
retries, except for 'acceptance' tests, including
'acceptanceInstall' tests.

Resolves: #96999
Releases: main, 11.5
Change-Id: I570c7fc6c61fe55ab5f60b51c04161f935876d24
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/73647


Tested-by: core-ci <typo3@b13.com>
Tested-by: Stefan Bürk <stefan@buerk.tech>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Casual "pre-merge" patches (pushing a patch) to CI
gives us two stages: An "early" stage checking CGL
and composer basics, plus the "main" stage checking
everything else.

The goal with this approach is to return early on
simple errors like CGL violations without wasting
additional ressources on other things when those
pre-conditions fail.

In practice, an early "cgl broken" pipeline shadowes
possible further issues and extends runtime of
single builds since 'main' is only run after 'early'
has been successful.

The patch merges all "pre-merge" jobs into one
stage. This means more jobs are executed in parallel,
but we hope the infrastructure can cope with that.
If that works out, CI should answer more quickly
on single patches.

Resolves: #96841
Releases: main, 11.5
Change-Id: I986992f486b05845621f35228ae01b673368ba95
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/73436


Tested-by: core-ci <typo3@b13.com>
Tested-by: Stefan Bürk <stefan@buerk.tech>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Change-Id: I13ca3c60a040c15aca8e259e7c65d33682f37c44
Resolves: #96131
Releases: main
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72349


Tested-by: Benni Mack <benni@typo3.org>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Benni Mack <benni@typo3.org>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Switching to 20.10 gives us a feature we need
in runTests.sh.

Resolves: #96013
Related: #94932
Related: #95999
Releases: master, 10.4
Change-Id: I7eef83c130b76cab84dff7d595b7807d34b22ce7
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72212


Tested-by: core-ci <typo3@b13.com>
Tested-by: Simon Gilli <typo3@gilbertsoft.org>
Tested-by: Stefan Bürk <stefan@buerk.tech>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Simon Gilli <typo3@gilbertsoft.org>
Reviewed-by: Stefan Bürk <stefan@buerk.tech>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

This gitlab-runner flag invokes a 'git clean' command
at the end of each job run, after uploading caches, to
remove any files that have been created by the job.

Until now, this cleanup has been done at the start
of the next job only, so all this temp data - around
half a gig to a gig per job - has been lying around
in the gitlab-runner docker cache volumes.

This is super useful since our runners work with a
ram driven var/lib/docker, this cleanup significantly
reduces ram usage on the runners.

Change-Id: I301d9ea24fab2c4c2202bd57c8e5ed6fd5f05fc0
Resolves: #95665
Releases: master, 10.4
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/71807


Tested-by: Jochen <rothjochen@gmail.com>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Jochen <rothjochen@gmail.com>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

A new flag adds more "timing bubbles" in job runs.
It helps finding slow commands more easily and
is useful with jobs that execute multiple runTests.sh
commands after each other, like the "integrity job".

Change-Id: I7427eaadb6aaaa231164d945b36fd2ba1bacd0b6
Resolves: #95664
Releases: master, 10.4
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/71801


Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Since moving to gitlab, script checkFilePermissions.sh
was inactive due to a gitlab detail that leads to
funny permissions of cloned files.

This can be suppressed with a gitlab feature toggle.

The patch activates that toggle, activates the test
script and fixes a couple of files that slipped in
with broken permissions meanwhile.

Change-Id: I2b0fdca282dbc39fe230b33910a690118a21b74e
Resolves: #95194
Releases: master, 10.4, 9.5
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/71033


Tested-by: core-ci <typo3@b13.com>
Tested-by: Wouter Wolters <typo3@wouterwolters.nl>
Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Wouter Wolters <typo3@wouterwolters.nl>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Now that we have frontend tests in the Backend suite the
name Backend for acceptance suite does not fit anymore.

So the suite name was changed from Backend to Application.

Resolves: #94864
Releases: master
Change-Id: Ib52053f6dbbfad9b7fef1ff11e81110cffa010b2
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/70550


Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>

There is no valid reason anymore to separate functional
test stages into 2 runs, so they get merged again.

Resolves: #94064
Releases: master, 10.4
Change-Id: I02385b2332b62ac9f637442b2500b214bd66b357
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/69019


Tested-by: core-ci <typo3@b13.com>
Tested-by: Daniel Goerz <daniel.goerz@posteo.de>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Reviewed-by: Daniel Goerz <daniel.goerz@posteo.de>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>

gerrit voting and slack nightly reporting is now done
with a gitlab webhook instead of a CI stage.

Change-Id: I6498c41229cb3b6a9178d2a91dbbfbb657034cce
Resolves: #94035
Releases: master, 10.4, 9.5
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/68956


Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Jobs are sometimes stuck, for instance when they run
into infinite loops. With the default job timeout of
one hour and a default retry, it takes two hours until
those report negative.
Jobs are usually done in less than 15 minutes. We now
set the timeout to generous 30 minutes, a pipeline with
a stuck job will thus report after one hour, latest.

Resolves: #94033
Related: #94001
Releases: master, 10.4, 9.5
Change-Id: I80c26783f5ff0a5900c849d84f57c77e22ca4f63
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/68954


Tested-by: core-ci <typo3@b13.com>
Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: Oliver Bartsch <bo@cedev.de>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Reviewed-by: Oliver Bartsch <bo@cedev.de>

The acceptance test suite for the stand alone install tool can
easily coexist with the other backend related tests, there is
no good reason for the overhead of a separate suite. So it gets
merged into the backend suite.

Resolves: #93621
Releases: master, 10.4
Change-Id: I460596985cd992c4f7d59df989c7992590e479d8
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/68158


Tested-by: core-ci <typo3@b13.com>
Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: Andreas Fernandez <a.fernandez@scripting-base.de>
Tested-by: Richard Haeser <richard@richardhaeser.com>
Reviewed-by: Andreas Fernandez <a.fernandez@scripting-base.de>
Reviewed-by: Richard Haeser <richard@richardhaeser.com>

The Pagetree suite for acceptance tests was set up as a separate
suite in the beginning due to a necessary fixture set of pages,
that all the other tests don't need.
The overhead of having separate suite is higher than the additional
fixture set, so the suites are merged.

Resolves: #93614
Releases: master, 10.4
Change-Id: I70a3f1dd68bb07c0dc505eaadb3d240e81639add
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/68151


Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Richard Haeser <richard@richardhaeser.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Richard Haeser <richard@richardhaeser.com>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Each jobs will have one more run in case it fails in the first.
This will prevent pipeline failures due to recoverable test
run failures.

Resolves: #93603
Releases: master, 10.4, 9.5
Change-Id: Ia054973d1f2a336727ef9d483041f1f48147c375
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/68140


Tested-by: Simon Gilli <typo3@gilbertsoft.org>
Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Simon Gilli <typo3@gilbertsoft.org>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Resolves: #93592
Releases: master, 10.4, 9.5
Change-Id: If851b9264d7d9e3b2a18df49d2c445f034aa5a80
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/68133


Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

* Acceptance tests in nightly run can use just one out of our
  supported databases, that should be enough.

* Some reorderings to reflect order in gitlab-ci better

* Minor issues and cleanups

Releases: master
Resolves: #93456
Change-Id: I67355f139e24ec9c287ab3b753059a342b387700
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/67660


Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: core-ci <typo3@b13.com>
Tested-by: Oliver Bartsch <bo@cedev.de>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Oliver Bartsch <bo@cedev.de>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

Resolves: #93434
Releases: master
Change-Id: I9d6918f5bbfc6f12d09b1459405e6749f04b2c0c
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/67641


Tested-by: core-ci <typo3@b13.com>
Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: Richard Haeser <richard@richardhaeser.com>
Tested-by: Oliver Bartsch <bo@cedev.de>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Richard Haeser <richard@richardhaeser.com>
Reviewed-by: Oliver Bartsch <bo@cedev.de>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>

A first Build/gitlab-ci.yml for master. Core v10 and
v9 will have dedicated patches.

This specifies jobs executed via gitlab CI for master
pre-merge "a patch has been pushed" and nightly builds.

This uses a new testing infrastructure defined by
https://gitlab.typo3.org/core-testing/testing-infrastructure
to execute test suites.

Resolves: #93406
Releases: master
Change-Id: I377ac59478925b789f63871c8fa6c3b2d5b2a839
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/67626


Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: Simon Gilli <typo3@gilbertsoft.org>
Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Reviewed-by: Simon Gilli <typo3@gilbertsoft.org>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>