[BUGFIX] No "update storage index" FAL task fail with too many records
The Indexer builds a large array of all actual files on a storage (identifiedFileUids). If many files exists, this array can get very large. This array was then passed to a QueryBuilder to fetch all records NOT IN that array. Since a NOT IN query is passed as a string to the database, it can exceed the string size allowed in a query, making the whole task fail. Since a NOT IN query cannot be chunked easily, the whole logic has been adapted in a different way. Instead of fetching a restricted list of database records, all records are fetched and iterated. Even with a million of sys_file_records of a single (!) storage this will perform alright, and be within practical usage scenarios. Each database record is then checked for a match in the large array of known records, and then execute the same logic as before. To benchmark the implications, the following test was run: Baseline: * sys_file with 50.736 entries * 16.912 marked as...
Please register or sign in to comment