Skip to content
Snippets Groups Projects
Select Git revision
0 result
Search by author
  • Any Author
0 authors
  1. May 11, 2024
  3. May 10, 2024
  5. May 09, 2024
  7. May 08, 2024
  9. May 07, 2024
  11. May 06, 2024
  13. May 05, 2024
  15. May 03, 2024
  17. May 02, 2024
  19. May 01, 2024
  21. Apr 30, 2024
  23. Apr 29, 2024
    • Jasmina Ließmann's avatar
      [BUGFIX] Add missing hover style to link in "Access" backend module · 1d86d5b4
      Jasmina Ließmann authored 
      The title of the linked page was missing a hover style, which is now
ensured with this patch. An appropriate link title is also added.

Resolves: #103715
Releases: main, 12.4
Change-Id: I4ec60ef43fd7ced6cdc5ede88445931a37e12c3a
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84011


Tested-by: default avatarcore-ci <typo3@b13.com>
Tested-by: default avatarJasmina Ließmann <minapokhalo+typo3@gmail.com>
Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: default avatarJasmina Ließmann <minapokhalo+typo3@gmail.com>
      1d86d5b4
    • Christian Kuhn's avatar
      [TASK] Use refindex in RootlineUtility, drop addRootLineFields · c124e904
      Christian Kuhn authored 
      tl;dr:

This patch switches from using RelationHandler to ReferenceIndex
in (mostly) frontend request related RootlineUtility. It is the
first patch actively using ReferenceIndex in frontend and has
the potential to significantly reduce database query load on
not-yet-cached pages in frontend, while creating better results
at the same time. The patch obsoletes
$GLOBALS['TYPO3_CONF_VARS']['FE']['addRootLineFields']. This
is just the tip of the iceberg from a refactoring chain that has
been prepared over a period of about three years already.

The long story:

Rendering times of complex pages in the frontend in general scale
linearly with the number of executed database queries (plus some
offset), if no extension needs extensive calculation on the PHP side,
even if the database is server-side local and not a remote instance,
which adds additional TCP round-trip times per query.
With thousands of queries being triggered by RootlineUtility, this
area is the most promising code path to optimize when response times
of not-fully-cached pages should be reduced.

RootlineUtility is *the* main performance bottleneck in
non-trivial TYPO3 instance frontend rendering: When rendering menus,
each single entry needs to know its page row details and its path in
the rootline, using RootlineUtility. This can quickly lead to
thousands of DB queries with bigger instances on cold caches,
catapulting the rendering times from milliseconds to seconds. This is
worse when dealing with language overlays and workspaces.

RootlineUtility does two main things:
It first resolves the rootline of the given pages-uid. This is
an expensive operation since it needs a query per page level.
We'll change this towards a recursive query soon. Afterwards, it
resolves some relations attached to the pages. This is - in default
instances - at least the 'media' field. This is done using
class RelationHandler per single field in question, which fires
at least one query per page row and per field.

As such, beside the expensive rootline determination itself, we also
have an expensive operation to resolve relations of each pages row.

Resolving relations attached to pages in a more performant way
is what this patch is about.

The list of relations being resolved can be extended using
$GLOBALS['TYPO3_CONF_VARS']['FE']['addRootLineFields']. By default,
only field 'media' is resolved, resulting in one additional query
per resolved page row. Instances tend to add more relations to
pages, and when adding these to 'addRootLineFields', they trigger
additional RelationHandler queries by RootlineUtility. Also,
RootlineUtility *ignores* 'addRootLineFields' when resolving
relations when requesting non-default (sys_language_uid != 0) pages,
and simply resolves *all* possible relation fields. This includes
'category' and probably some of the TCA fields ext:seo adds to pages
TCA. RelationHandler needs *at least* one query per field, more with
localizations and even more when rendering a workspaces preview.

This situation makes relation resolving of page rows in
RootlineUtility more expensive than resolving the rootline itself.
One option would be to skip resolving page relations in
RootlineUtility, but that wouldn't help us much when instances
actually need this data (which is likely).
A better solution is to supply relations within pages in a more
performant way, ready for TypoScript and extensions to use.

The patch switches from using RelationHandler within
RootlineUtility to query the ReferenceIndex instead: sys_refindex
contains all fields relevant for frontend since #103748. It allows
querying *all* relations of *all* page fields with just *one* query.

This allows us to remove 'addRootLineFields' by just always
resolving all relations with close to no overhead, and turns into
an advantage for integrators: One detail less to take care of.

The patch establishes a nifty - but very well indexed - query within
RootlineUtility to fetch relations. With 'categories' (an MM foreign
relation) being attached to pages by default, it is a UNION query of
two SELECT queries, with their result being sorted out in a rather
effective result handling loop.

The patch comes with a todo list within RootlineUtility, but in
general, the result list is *better* than the result of the previous
single field based RelationHandler solution. This is verified by the
previously establish test data sets:
* hidden, starttime, endtime are resolved.
* Workspace deleted placeholder rows are removed automatically.

There are a couple of details we may want to improve with further
v13 patches, but in general, this new approach should work well
already.

Future:

* This patch is a base to unlock a series of further performance
  related changes: When CTE's on RootlineUtility are established, it
  will allow fetching relations of not only one page, but of multiple
  pages with one query. Combined with a couple of other changes, the
  query load in frontend will ultimately collapse.

* The patch is the base for ReferenceIndex usage in the frontend:
  We can potentially remove the doomed DatabaseQueryProcessor and
  other shenanigans for tt_content within the frontend rendering
  chain by always providing "correct" relation information
  automatically without overhead. Other pending patches are already
  preparing a central place taking care of tt_content for its
  relations to be resolved in a similar effective way.

* Resolving relations of all fields attached to page rows voids the
  'parent count' values of these fields. Those are a different
  de-normalization of relations "if it's not 0, we have some". With
  RootlineUtility dropping 'addRootLineFields' and always resolving
  relations, we can finally start dropping these 'count' fields, can
  release DataHandler from updating these fields, and can remove
  them from DB altogether.

Numbers:

There was a recent blog post by Julian [1] looking at query load of
bigger menus. With measurements for this patch, we confirm the
general numbers Julian came up with in his test setup. Thank to the
published test setup at [2], we ported it to TYPO3 v13, and slightly
extended it to measure query load when all pages are localized, and
we're performing a request to a non-default language.

* The main query count on a "cold" instance calling default language
  does *not* change with this patch. The reason is, default TYPO3
  includes only 'media' as relation that has a 'foreign_field' in
  RootlineUtility, which leads to only one query, which is now
  substituted by the new refindex query. The advantage of the new
  solution provided by the patch is to resolve *all* relation with
  the same query load.

* When calling a localized page, the query load is at ~10200 queries
  on a non-cached-page frontend request. This shrinks to ~7900 with
  the patch being applied. That's a reduction of 1/5 by this patch,
  with further options up our sleeves.

[1] https://www.in2code.de/aktuelles/menue-vergleich-der-techniken/
[2] https://github.com/julianhofmann/typo3-menu-comparison

Resolves: #103752
Related: #103748
Related: #103735
Related: #103710
Related: #103680
Related: #103599
Related: #103598
Change-Id: Ic7e152366dc2d4312ca55035c364db9ac02fabee
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/83729


Tested-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: default avatarAndré Buchmann <andy.schliesser@gmail.com>
Reviewed-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: default avatarBenni Mack <benni@typo3.org>
Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarAndré Buchmann <andy.schliesser@gmail.com>
Reviewed-by: default avatarBenni Mack <benni@typo3.org>
Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
Tested-by: default avatarcore-ci <typo3@b13.com>
Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      c124e904
  25. Apr 28, 2024