Skip to content
Snippets Groups Projects
  1. May 03, 2024
  2. May 02, 2024
  3. May 01, 2024
  4. Apr 30, 2024
  5. Apr 29, 2024
    • Jasmina Ließmann's avatar
      [BUGFIX] Add missing hover style to link in "Access" backend module · 1d86d5b4
      Jasmina Ließmann authored
      The title of the linked page was missing a hover style, which is now
      ensured with this patch. An appropriate link title is also added.
      
      Resolves: #103715
      Releases: main, 12.4
      Change-Id: I4ec60ef43fd7ced6cdc5ede88445931a37e12c3a
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84011
      
      
      Tested-by: default avatarcore-ci <typo3@b13.com>
      Tested-by: default avatarJasmina Ließmann <minapokhalo+typo3@gmail.com>
      Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
      Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: default avatarJasmina Ließmann <minapokhalo+typo3@gmail.com>
      1d86d5b4
    • Christian Kuhn's avatar
      [TASK] Use refindex in RootlineUtility, drop addRootLineFields · c124e904
      Christian Kuhn authored
      tl;dr:
      
      This patch switches from using RelationHandler to ReferenceIndex
      in (mostly) frontend request related RootlineUtility. It is the
      first patch actively using ReferenceIndex in frontend and has
      the potential to significantly reduce database query load on
      not-yet-cached pages in frontend, while creating better results
      at the same time. The patch obsoletes
      $GLOBALS['TYPO3_CONF_VARS']['FE']['addRootLineFields']. This
      is just the tip of the iceberg from a refactoring chain that has
      been prepared over a period of about three years already.
      
      The long story:
      
      Rendering times of complex pages in the frontend in general scale
      linearly with the number of executed database queries (plus some
      offset), if no extension needs extensive calculation on the PHP side,
      even if the database is server-side local and not a remote instance,
      which adds additional TCP round-trip times per query.
      With thousands of queries being triggered by RootlineUtility, this
      area is the most promising code path to optimize when response times
      of not-fully-cached pages should be reduced.
      
      RootlineUtility is *the* main performance bottleneck in
      non-trivial TYPO3 instance frontend rendering: When rendering menus,
      each single entry needs to know its page row details and its path in
      the rootline, using RootlineUtility. This can quickly lead to
      thousands of DB queries with bigger instances on cold caches,
      catapulting the rendering times from milliseconds to seconds. This is
      worse when dealing with language overlays and workspaces.
      
      RootlineUtility does two main things:
      It first resolves the rootline of the given pages-uid. This is
      an expensive operation since it needs a query per page level.
      We'll change this towards a recursive query soon. Afterwards, it
      resolves some relations attached to the pages. This is - in default
      instances - at least the 'media' field. This is done using
      class RelationHandler per single field in question, which fires
      at least one query per page row and per field.
      
      As such, beside the expensive rootline determination itself, we also
      have an expensive operation to resolve relations of each pages row.
      
      Resolving relations attached to pages in a more performant way
      is what this patch is about.
      
      The list of relations being resolved can be extended using
      $GLOBALS['TYPO3_CONF_VARS']['FE']['addRootLineFields']. By default,
      only field 'media' is resolved, resulting in one additional query
      per resolved page row. Instances tend to add more relations to
      pages, and when adding these to 'addRootLineFields', they trigger
      additional RelationHandler queries by RootlineUtility. Also,
      RootlineUtility *ignores* 'addRootLineFields' when resolving
      relations when requesting non-default (sys_language_uid != 0) pages,
      and simply resolves *all* possible relation fields. This includes
      'category' and probably some of the TCA fields ext:seo adds to pages
      TCA. RelationHandler needs *at least* one query per field, more with
      localizations and even more when rendering a workspaces preview.
      
      This situation makes relation resolving of page rows in
      RootlineUtility more expensive than resolving the rootline itself.
      One option would be to skip resolving page relations in
      RootlineUtility, but that wouldn't help us much when instances
      actually need this data (which is likely).
      A better solution is to supply relations within pages in a more
      performant way, ready for TypoScript and extensions to use.
      
      The patch switches from using RelationHandler within
      RootlineUtility to query the ReferenceIndex instead: sys_refindex
      contains all fields relevant for frontend since #103748. It allows
      querying *all* relations of *all* page fields with just *one* query.
      
      This allows us to remove 'addRootLineFields' by just always
      resolving all relations with close to no overhead, and turns into
      an advantage for integrators: One detail less to take care of.
      
      The patch establishes a nifty - but very well indexed - query within
      RootlineUtility to fetch relations. With 'categories' (an MM foreign
      relation) being attached to pages by default, it is a UNION query of
      two SELECT queries, with their result being sorted out in a rather
      effective result handling loop.
      
      The patch comes with a todo list within RootlineUtility, but in
      general, the result list is *better* than the result of the previous
      single field based RelationHandler solution. This is verified by the
      previously establish test data sets:
      * hidden, starttime, endtime are resolved.
      * Workspace deleted placeholder rows are removed automatically.
      
      There are a couple of details we may want to improve with further
      v13 patches, but in general, this new approach should work well
      already.
      
      Future:
      
      * This patch is a base to unlock a series of further performance
        related changes: When CTE's on RootlineUtility are established, it
        will allow fetching relations of not only one page, but of multiple
        pages with one query. Combined with a couple of other changes, the
        query load in frontend will ultimately collapse.
      
      * The patch is the base for ReferenceIndex usage in the frontend:
        We can potentially remove the doomed DatabaseQueryProcessor and
        other shenanigans for tt_content within the frontend rendering
        chain by always providing "correct" relation information
        automatically without overhead. Other pending patches are already
        preparing a central place taking care of tt_content for its
        relations to be resolved in a similar effective way.
      
      * Resolving relations of all fields attached to page rows voids the
        'parent count' values of these fields. Those are a different
        de-normalization of relations "if it's not 0, we have some". With
        RootlineUtility dropping 'addRootLineFields' and always resolving
        relations, we can finally start dropping these 'count' fields, can
        release DataHandler from updating these fields, and can remove
        them from DB altogether.
      
      Numbers:
      
      There was a recent blog post by Julian [1] looking at query load of
      bigger menus. With measurements for this patch, we confirm the
      general numbers Julian came up with in his test setup. Thank to the
      published test setup at [2], we ported it to TYPO3 v13, and slightly
      extended it to measure query load when all pages are localized, and
      we're performing a request to a non-default language.
      
      * The main query count on a "cold" instance calling default language
        does *not* change with this patch. The reason is, default TYPO3
        includes only 'media' as relation that has a 'foreign_field' in
        RootlineUtility, which leads to only one query, which is now
        substituted by the new refindex query. The advantage of the new
        solution provided by the patch is to resolve *all* relation with
        the same query load.
      
      * When calling a localized page, the query load is at ~10200 queries
        on a non-cached-page frontend request. This shrinks to ~7900 with
        the patch being applied. That's a reduction of 1/5 by this patch,
        with further options up our sleeves.
      
      [1] https://www.in2code.de/aktuelles/menue-vergleich-der-techniken/
      [2] https://github.com/julianhofmann/typo3-menu-comparison
      
      Resolves: #103752
      Related: #103748
      Related: #103735
      Related: #103710
      Related: #103680
      Related: #103599
      Related: #103598
      Change-Id: Ic7e152366dc2d4312ca55035c364db9ac02fabee
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/83729
      
      
      Tested-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
      Tested-by: default avatarAndré Buchmann <andy.schliesser@gmail.com>
      Reviewed-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
      Tested-by: default avatarBenni Mack <benni@typo3.org>
      Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: default avatarAndré Buchmann <andy.schliesser@gmail.com>
      Reviewed-by: default avatarBenni Mack <benni@typo3.org>
      Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      Tested-by: default avatarcore-ci <typo3@b13.com>
      Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      c124e904
  6. Apr 28, 2024
    • Torben Hansen's avatar
      [BUGFIX] Avoid undefined array key warnings in SuggestWizardController · 5143487f
      Torben Hansen authored
      When an editor uses the suggest wizard, undefined array key warnings
      can occur, when certain array keys have not been defined.
      
      This change adds fallbacks for undefined array keys using the
      null coalescing operator. Additionally, the function
      `currentBackendUserMayAccessTable` is now covered with unit tests.
      
      Resolves: #103721
      Releases: main, 12.4
      Change-Id: I2b1a8af0883f7d34dde38f6f838df080f34ea87c
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84074
      
      
      Tested-by: default avatarcore-ci <typo3@b13.com>
      Reviewed-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
      Tested-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
      Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
      5143487f
    • Chris Müller's avatar
      [DOCS] Fix typo in changelog to TypoScript provider for sites and sets · 9ff2b1b0
      Chris Müller authored
      Resolves: #103755
      Related: #103675
      Releases: main
      Change-Id: Ic94dc6bbff56f09233fec1b7cda42091d07dad2f
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84077
      
      
      Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
      Tested-by: default avatarcore-ci <typo3@b13.com>
      9ff2b1b0
    • Andreas Kienast's avatar
      [BUGFIX] Migrate Bootstrap progress bars to bootstrap 5.3 · 9fafcec2
      Andreas Kienast authored
      With the update of Bootstrap to v5.3, the implementation of progress
      bars has changed for accessibility reasons [1].
      
      This commit updates all usages of progress bars to the new markup.
      
      [1] https://getbootstrap.com/docs/5.3/migration/#progress-bars
      
      Resolves: #103718
      Releases: main, 12.4
      Change-Id: I208e3561b4943e382caa09e23e75d350bd702a6a
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84014
      
      
      Tested-by: default avatarBenjamin Franzke <ben@bnf.dev>
      Tested-by: default avatarAndreas Kienast <a.fernandez@scripting-base.de>
      Reviewed-by: default avatarBenjamin Franzke <ben@bnf.dev>
      Tested-by: default avatarAndreas Nedbal <andy@pixelde.su>
      Reviewed-by: default avatarAndreas Kienast <a.fernandez@scripting-base.de>
      Tested-by: default avatarcore-ci <typo3@b13.com>
      Reviewed-by: default avatarAndreas Nedbal <andy@pixelde.su>
      9fafcec2
    • Torben Hansen's avatar
      [TASK] Remove Flow framework related todos in Extbase · d47e7559
      Torben Hansen authored
      3 todos in Extbase refer to aligning classes or functionality
      closer to the Flow framework. Those todos have been added
      11 years ago with #47251.
      
      Since there are no plans to align Extbase closer to the Flow
      Framework functionality, the todos can safely be removed.
      
      Resolves: #103753
      Releases: main
      Change-Id: I9e81c0f3c27a4e690587f7f6a50a21376ba8ca2e
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84076
      
      
      Reviewed-by: default avatarNikita Hovratov <nikita.h@live.de>
      Tested-by: default avatarNikita Hovratov <nikita.h@live.de>
      Tested-by: default avatarcore-ci <typo3@b13.com>
      Tested-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
      Reviewed-by: default avatarOliver Klee <typo3-coding@oliverklee.de>
      d47e7559
    • Christian Kuhn's avatar
      [TASK] Enrich ReferenceIndex with more relation data · 28aac3c6
      Christian Kuhn authored
      ReferenceIndex (table sys_refindex) has two main jobs:
      * It registers "soft" relations, for instance when a
        bodytext sets a link in its text string. Similar things
        are done for relations in anonymous flex form strings.
      * It is a registry of DB relations attached to TCA based
        table records.
      
      This patch is about the DB relation part of refindex.
      
      When a TCA record needs to know all its database relations
      attached to it, it can either ask the (clumsy)
      RelationHandler for relations of each field, which then
      queries the foreign table for relations, OR it could ask
      the refindex for all relations with a single query. It
      is possible to do this for multiple records in one query.
      
      Refindex has significant test coverage within DataHandler:
      With over a thousand functional tests validating the
      refindex, we can be confident in the accuracy of the
      functionality. This coverage has been in place and
      continuously improved since TYPO3 v11.
      
      This de-normalized nature of having a single table that
      tracks all TCA DB relations of all TCA tables is a powerful
      structure that allows us to query relations in a performant
      way. This is currently only leveraged in a couple of places
      in the backend, though.
      
      Integrating refindex within the frontend is promising to
      alleviate the query load on not-yet-cached pages, though:
      There are various places that tend to hammer the database
      with hundreds or thousand of queries to look up relations,
      RootlineUtility in particular is very demanding.
      
      Refindex however does not carry relation details important
      in the frontend: It does not know about hidden, starttime
      and endtime restrictions applied to single relations.
      
      The patch changes this by adding a series of columns to
      sys_refindex, and adapts class ReferenceIndex to take care
      of them.
      
      Adding fields to sys_refindex has one drawback:
      Primary key 'hash' is a hash of all columns. Adding
      columns invalidates all existing hashes, refindex needs
      to be rebuild. This is why the patch is so huge: All tests
      checking refindex integrity need new rows. It also means
      instances need to update refindex. The patch dedicates
      an Important.rst to this.
      
      With forcing a full re-index for everyone, the patch takes
      the opportunity to use xxh128 over md5 for the hash, which
      is a significantly quicker hash algorithm with good
      collision protection, and with the same length of 128
      bit (128 bit = 16 bytes = 16 hex numbers = 32 chars).
      
      The patch also adds more fine grained table column and
      index definitions: Two changed indexes are designed to help
      queries having the typical query signature, and queries can
      be crafted to use large parts of the combined indexes.
      
      Another detail is that 'endtime' defaults to a high value:
      When a casual endtime-aware TCA record has no endtime set,
      it's column value is 0. This requires an OR in the query.
      refindex de-normalizes this to a high value instead, to
      always query rows with a "greater than now" where clause.
      
      The patch reveals a bug within MM when resorting rows. A
      single functional test is disabled for now. The situation
      will be fixed with another patch.
      
      sys_refindex contains columns for the "left" / "local" side,
      and columns for the "right" / "foreign" side. Foreign side
      columns are prefixed with "ref_". A quick rundown of
      sys_refindex table columns:
      
      hash:
      primaray key, xxh128 hash of the other column values
      
      tablename & ref_table:
      names of the involved tables. tablename is "_STRING" when
      dealing with a softref row.
      
      recuid & ref_uid:
      uids of the involved table rows.
      
      field & (new)ref_field:
      names of the involved table fields.
      
      sorting & ref_sorting:
      keep relations in a deterministic order.
      
      (new)hidden & (new)ref_hidden:
      int '1' if the table is TCA hidden aware and the row is hidden.
      
      (new)starttime & (new)ref_starttime:
      starttime timestamp of the row if the table is TCA starttime
      aware and a value is set, else '0'.
      
      (new)endtime & (new)ref_endtime:
      endtime timestamp of the row if the table is TCA endtime aware
      and a value is set, else 2^31-1 (year 2038) since postgres 32-bit
      int is always signed. Could be extended later.
      
      workspace:
      calculated workspace-uid this refindex row is relevant in. With
      MM relations, there can be rows with workspace-uid 0 on one side
      and workspace-uid 1 on the other side, with the sys_refindex
      row being valid for workspace-uid 1.
      
      (new)t3ver_state & (new)ref_t3ver_state:
      workspace versionState value of the table row, default int 0.
      Used to filter especially deleted placeholder rows (int 2).
      
      flexpointer, softref_key, softref_id, ref_string:
      softref and flex form path details fields.
      
      Resolves: #103748
      Related: #103735
      Related: #103680
      Related: #103655
      Releases: main
      Change-Id: I276eef06da4bbe5bd51d90fc850e27ebe6ff2130
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/84050
      
      
      Tested-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
      Tested-by: default avatarAnja Leichsenring <aleichsenring@ab-softlab.de>
      Tested-by: default avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: default avatarStefan Bürk <stefan@buerk.tech>
      Tested-by: default avatarcore-ci <typo3@b13.com>
      Reviewed-by: default avatarChristian Kuhn <lolli@schwarzbu.ch>
      28aac3c6
  7. Apr 27, 2024
  8. Apr 26, 2024
  9. Apr 25, 2024