Skip to content
Snippets Groups Projects
Commit 390092d8 authored by Garvin Hicking's avatar Garvin Hicking Committed by Stefan Bürk
Browse files

[BUGFIX] No "update storage index" FAL task fail with too many records

The Indexer builds a large array of all actual files on a storage
(identifiedFileUids). If many files exists, this array can get
very large.

This array was then passed to a QueryBuilder to fetch all
records NOT IN that array. Since a NOT IN query is passed as a
string to the database, it can exceed the string size allowed
in a query, making the whole task fail.

Since a NOT IN query cannot be chunked easily, the whole
logic has been adapted in a different way.

Instead of fetching a restricted list of database records,
all records are fetched and iterated. Even with a million
of sys_file_records of a single (!) storage this will
perform alright, and be within practical usage scenarios.

Each database record is then checked for a match in the
large array of known records, and then execute the
same logic as before.

To benchmark the implications, the following test was run:

Baseline:

* sys_file with 50.736 entries
  * 16.912 marked as...
parent 5e2a9d4e
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment