Skip to content
Snippets Groups Projects
Select Git revision
  • 12.4
  • main default
  • 11.5
  • 13.3
  • l10n_main
  • 13.1
  • 13.0
  • l10n_12.4
  • 10.4
  • 12.1
  • 9.5
  • 11.3
  • master
  • 11.1
  • TYPO3_8-7
  • 10.2
  • 9.3
  • 9.2
  • TYPO3_7-6
  • TYPO3_7-0
  • v13.3.1
  • v12.4.21
  • v11.5.40
  • v13.3.0
  • v12.4.20
  • v12.4.19
  • v12.4.18
  • v11.5.39
  • v12.4.17
  • v13.2.1
  • v13.2.0
  • v12.4.16
  • v11.5.38
  • v13.1.1
  • v12.4.15
  • v11.5.37
  • v13.1.0
  • v12.4.14
  • v12.4.13
  • v12.4.12
40 results
Search by author
  • Any Author
  • Daniel Hürtgen Daniel Hürtgen higidi
  • Jenkins Jenkins jenkins
  • **** **** project_53_bot
3 authors
  1. May 05, 2024
  3. May 03, 2024
  5. May 02, 2024
  7. May 01, 2024
  9. Apr 30, 2024
  11. Apr 29, 2024
    • Jasmina Ließmann's avatar
      [BUGFIX] Add missing hover style to link in "Access" backend module · 1d86d5b4
      Jasmina Ließmann authored 
      The title of the linked page was missing a hover style, which is now
ensured with this patch. An appropriate link title is also added.

Resolves: #103715
Releases: main, 12.4
Change-Id: I4ec60ef43fd7ced6cdc5ede88445931a37e12c3a
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84011


Tested-by: default avatarcore-ci <typo3@b13.com>
Tested-by: default avatarJasmina Ließmann <minapokhalo+typo3@gmail.com>
Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: default avatarJasmina Ließmann <minapokhalo+typo3@gmail.com>
      1d86d5b4
    • Christian Kuhn's avatar
      [TASK] Use refindex in RootlineUtility, drop addRootLineFields · c124e904
      Christian Kuhn authored 
      tl;dr:

This patch switches from using RelationHandler to ReferenceIndex
in (mostly) frontend request related RootlineUtility. It is the
first patch actively using ReferenceIndex in frontend and has
the potential to significantly reduce database query load on
not-yet-cached pages in frontend, while creating better results
at the same time. The patch obsoletes
$GLOBALS['TYPO3_CONF_VARS']['FE']['addRootLineFields']. This
is just the tip of the iceberg from a refactoring chain that has
been prepared over a period of about three years already.

The long story:

Rendering times of complex pages in the frontend in general scale
linearly with the number of executed database queries (plus some
offset), if no extension needs extensive calculation on the PHP side,
even if the database is server-side local and not a remote instance,
which adds additional TCP round-trip times per query.
With thousands of queries being triggered by RootlineUtility, this
area is the most promising code path to optimize when response times
of not-fully-cached pages should be reduced.

RootlineUtility is *the* main performance bottleneck in
non-trivial TYPO3 instance frontend rendering: When rendering menus,
each single entry needs to know its page row details and its path in
the rootline, using RootlineUtility. This can quickly lead to
thousands of DB queries with bigger instances on cold caches,
catapulting the rendering times from milliseconds to seconds. This is
worse when dealing with language overlays and workspaces.

RootlineUtility does two main things:
It first resolves the rootline of the given pages-uid. This is
an expensive operation since it needs a query per page level.
We'll change this towards a recursive query soon. Afterwards, it
resolves some relations attached to the pages. This is - in default
instances - at least the 'media' field. This is done using
class RelationHandler per single field in question, which fires
at least one query per page row and per field.

As such, beside the expensive rootline determination itself, we also
have an expensive operation to resolve relations of each pages row.

Resolving relations attached to pages in a more performant way
is what this patch is about.

The list of relations being resolved can be extended using
$GLOBALS['TYPO3_CONF_VARS']['FE']['addRootLineFields']. By default,
only field 'media' is resolved, resulting in one additional query
per resolved page row. Instances tend to add more relations to
pages, and when adding these to 'addRootLineFields', they trigger
additional RelationHandler queries by RootlineUtility. Also,
RootlineUtility *ignores* 'addRootLineFields' when resolving
relations when requesting non-default (sys_language_uid != 0) pages,
and simply resolves *all* possible relation fields. This includes
'category' and probably some of the TCA fields ext:seo adds to pages
TCA. RelationHandler needs *at least* one query per field, more with
localizations and even more when rendering a workspaces preview.

This situation makes relation resolving of page rows in
RootlineUtility more expensive than resolving the rootline itself.
One option would be to skip resolving page relations in
RootlineUtility, but that wouldn't help us much when instances
actually need this data (which is likely).
A better solution is to supply relations within pages in a more
performant way, ready for TypoScript and extensions to use.

The patch switches from using RelationHandler within
RootlineUtility to query the ReferenceIndex instead: sys_refindex
contains all fields relevant for frontend since #103748. It allows
querying *all* relations of *all* page fields with just *one* query.

This allows us to remove 'addRootLineFields' by just always
resolving all relations with close to no overhead, and turns into
an advantage for integrators: One detail less to take care of.

The patch establishes a nifty - but very well indexed - query within
RootlineUtility to fetch relations. With 'categories' (an MM foreign
relation) being attached to pages by default, it is a UNION query of
two SELECT queries, with their result being sorted out in a rather
effective result handling loop.

The patch comes with a todo list within RootlineUtility, but in
general, the result list is *better* than the result of the previous
single field based RelationHandler solution. This is verified by the
previously establish test data sets:
* hidden, starttime, endtime are resolved.
* Workspace deleted placeholder rows are removed automatically.

There are a couple of details we may want to improve with further
v13 patches, but in general, this new approach should work well
already.

Future:

* This patch is a base to unlock a series of further performance
  related changes: When CTE's on RootlineUtility are established, it
  will allow fetching relations of not only one page, but of multiple
  pages with one query. Combined with a couple of other changes, the
  query load in frontend will ultimately collapse.

* The patch is the base for ReferenceIndex usage in the frontend:
  We can potentially remove the doomed DatabaseQueryProcessor and
  other shenanigans for tt_content within the frontend rendering
  chain by always providing "correct" relation information
  automatically without overhead. Other pending patches are already
  preparing a central place taking care of tt_content for its
  relations to be resolved in a similar effective way.

* Resolving relations of all fields attached to page rows voids the
  'parent count' values of these fields. Those are a different
  de-normalization of relations "if it's not 0, we have some". With
  RootlineUtility dropping 'addRootLineFields' and always resolving
  relations, we can finally start dropping these 'count' fields, can
  release DataHandler from updating these fields, and can remove
  them from DB altogether.

Numbers:

There was a recent blog post by Julian [1] looking at query load of
bigger menus. With measurements for this patch, we confirm the
general numbers Julian came up with in his test setup. Thank to the
published test setup at [2], we ported it to TYPO3 v13, and slightly
extended it to measure query load when all pages are localized, and
we're performing a request to a non-default language.

* The main query count on a "cold" instance calling default language
  does *not* change with this patch. The reason is, default TYPO3
  includes only 'media' as relation that has a 'foreign_field' in
  RootlineUtility, which leads to only one query, which is now
  substituted by the new refindex query. The advantage of the new
  solution provided by the patch is to resolve *all* relation with
  the same query load.

* When calling a localized page, the query load is at ~10200 queries
  on a non-cached-page frontend request. This shrinks to ~7900 with
  the patch being applied. That's a reduction of 1/5 by this patch,
  with further options up our sleeves.

[1] https://www.in2code.de/aktuelles/menue-vergleich-der-techniken/
[2] https://github.com/julianhofmann/typo3-menu-comparison

Resolves: #103752
Related: #103748
Related: #103735
Related: #103710
Related: #103680
Related: #103599
Related: #103598
Change-Id: Ic7e152366dc2d4312ca55035c364db9ac02fabee
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/83729


Tested-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: default avatarAndré Buchmann <andy.schliesser@gmail.com>
Reviewed-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: default avatarBenni Mack <benni@typo3.org>
Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarAndré Buchmann <andy.schliesser@gmail.com>
Reviewed-by: default avatarBenni Mack <benni@typo3.org>
Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
Tested-by: default avatarcore-ci <typo3@b13.com>
Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      c124e904
  13. Apr 28, 2024
    • Torben Hansen's avatar
      [BUGFIX] Avoid undefined array key warnings in SuggestWizardController · 5143487f
      Torben Hansen authored 
      When an editor uses the suggest wizard, undefined array key warnings
can occur, when certain array keys have not been defined.

This change adds fallbacks for undefined array keys using the
null coalescing operator. Additionally, the function
`currentBackendUserMayAccessTable` is now covered with unit tests.

Resolves: #103721
Releases: main, 12.4
Change-Id: I2b1a8af0883f7d34dde38f6f838df080f34ea87c
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84074


Tested-by: default avatarcore-ci <typo3@b13.com>
Reviewed-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
Tested-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
      5143487f
    • Chris Müller's avatar
      [DOCS] Fix typo in changelog to TypoScript provider for sites and sets · 9ff2b1b0
      Chris Müller authored 
      Resolves: #103755
Related: #103675
Releases: main
Change-Id: Ic94dc6bbff56f09233fec1b7cda42091d07dad2f
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84077


Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
Tested-by: default avatarcore-ci <typo3@b13.com>
      9ff2b1b0
    • Andreas Kienast's avatar
      [BUGFIX] Migrate Bootstrap progress bars to bootstrap 5.3 · 9fafcec2
      Andreas Kienast authored 
      With the update of Bootstrap to v5.3, the implementation of progress
bars has changed for accessibility reasons [1].

This commit updates all usages of progress bars to the new markup.

[1] https://getbootstrap.com/docs/5.3/migration/#progress-bars

Resolves: #103718
Releases: main, 12.4
Change-Id: I208e3561b4943e382caa09e23e75d350bd702a6a
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84014


Tested-by: default avatarBenjamin Franzke <ben@bnf.dev>
Tested-by: default avatarAndreas Kienast <a.fernandez@scripting-base.de>
Reviewed-by: default avatarBenjamin Franzke <ben@bnf.dev>
Tested-by: default avatarAndreas Nedbal <andy@pixelde.su>
Reviewed-by: default avatarAndreas Kienast <a.fernandez@scripting-base.de>
Tested-by: default avatarcore-ci <typo3@b13.com>
Reviewed-by: default avatarAndreas Nedbal <andy@pixelde.su>
      9fafcec2
    • Torben Hansen's avatar
      [TASK] Remove Flow framework related todos in Extbase · d47e7559
      Torben Hansen authored 
      3 todos in Extbase refer to aligning classes or functionality
closer to the Flow framework. Those todos have been added
11 years ago with #47251.

Since there are no plans to align Extbase closer to the Flow
Framework functionality, the todos can safely be removed.

Resolves: #103753
Releases: main
Change-Id: I9e81c0f3c27a4e690587f7f6a50a21376ba8ca2e
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84076


Reviewed-by: default avatarNikita Hovratov <nikita.h@live.de>
Tested-by: default avatarNikita Hovratov <nikita.h@live.de>
Tested-by: default avatarcore-ci <typo3@b13.com>
Tested-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
Reviewed-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
      d47e7559
    • Christian Kuhn's avatar
      [TASK] Enrich ReferenceIndex with more relation data · 28aac3c6
      Christian Kuhn authored 
      ReferenceIndex (table sys_refindex) has two main jobs:
* It registers "soft" relations, for instance when a
  bodytext sets a link in its text string. Similar things
  are done for relations in anonymous flex form strings.
* It is a registry of DB relations attached to TCA based
  table records.

This patch is about the DB relation part of refindex.

When a TCA record needs to know all its database relations
attached to it, it can either ask the (clumsy)
RelationHandler for relations of each field, which then
queries the foreign table for relations, OR it could ask
the refindex for all relations with a single query. It
is possible to do this for multiple records in one query.

Refindex has significant test coverage within DataHandler:
With over a thousand functional tests validating the
refindex, we can be confident in the accuracy of the
functionality. This coverage has been in place and
continuously improved since TYPO3 v11.

This de-normalized nature of having a single table that
tracks all TCA DB relations of all TCA tables is a powerful
structure that allows us to query relations in a performant
way. This is currently only leveraged in a couple of places
in the backend, though.

Integrating refindex within the frontend is promising to
alleviate the query load on not-yet-cached pages, though:
There are various places that tend to hammer the database
with hundreds or thousand of queries to look up relations,
RootlineUtility in particular is very demanding.

Refindex however does not carry relation details important
in the frontend: It does not know about hidden, starttime
and endtime restrictions applied to single relations.

The patch changes this by adding a series of columns to
sys_refindex, and adapts class ReferenceIndex to take care
of them.

Adding fields to sys_refindex has one drawback:
Primary key 'hash' is a hash of all columns. Adding
columns invalidates all existing hashes, refindex needs
to be rebuild. This is why the patch is so huge: All tests
checking refindex integrity need new rows. It also means
instances need to update refindex. The patch dedicates
an Important.rst to this.

With forcing a full re-index for everyone, the patch takes
the opportunity to use xxh128 over md5 for the hash, which
is a significantly quicker hash algorithm with good
collision protection, and with the same length of 128
bit (128 bit = 16 bytes = 16 hex numbers = 32 chars).

The patch also adds more fine grained table column and
index definitions: Two changed indexes are designed to help
queries having the typical query signature, and queries can
be crafted to use large parts of the combined indexes.

Another detail is that 'endtime' defaults to a high value:
When a casual endtime-aware TCA record has no endtime set,
it's column value is 0. This requires an OR in the query.
refindex de-normalizes this to a high value instead, to
always query rows with a "greater than now" where clause.

The patch reveals a bug within MM when resorting rows. A
single functional test is disabled for now. The situation
will be fixed with another patch.

sys_refindex contains columns for the "left" / "local" side,
and columns for the "right" / "foreign" side. Foreign side
columns are prefixed with "ref_". A quick rundown of
sys_refindex table columns:

hash:
primaray key, xxh128 hash of the other column values

tablename & ref_table:
names of the involved tables. tablename is "_STRING" when
dealing with a softref row.

recuid & ref_uid:
uids of the involved table rows.

field & (new)ref_field:
names of the involved table fields.

sorting & ref_sorting:
keep relations in a deterministic order.

(new)hidden & (new)ref_hidden:
int '1' if the table is TCA hidden aware and the row is hidden.

(new)starttime & (new)ref_starttime:
starttime timestamp of the row if the table is TCA starttime
aware and a value is set, else '0'.

(new)endtime & (new)ref_endtime:
endtime timestamp of the row if the table is TCA endtime aware
and a value is set, else 2^31-1 (year 2038) since postgres 32-bit
int is always signed. Could be extended later.

workspace:
calculated workspace-uid this refindex row is relevant in. With
MM relations, there can be rows with workspace-uid 0 on one side
and workspace-uid 1 on the other side, with the sys_refindex
row being valid for workspace-uid 1.

(new)t3ver_state & (new)ref_t3ver_state:
workspace versionState value of the table row, default int 0.
Used to filter especially deleted placeholder rows (int 2).

flexpointer, softref_key, softref_id, ref_string:
softref and flex form path details fields.

Resolves: #103748
Related: #103735
Related: #103680
Related: #103655
Releases: main
Change-Id: I276eef06da4bbe5bd51d90fc850e27ebe6ff2130
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84050


Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
Tested-by: default avatarcore-ci <typo3@b13.com>
Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      28aac3c6
  15. Apr 27, 2024
  17. Apr 26, 2024