Revert "[BUGFIX] Avoid duplicates if ReferenceIndex is unable to finish"

This reverts commit c2a9726c
from review https://review.typo3.org/#/c/50803/
due to regression in #81320

Change-Id: I496e13ce9efe743a5607e12b46a3fdc90e2f17c2
Resolves: #81320
Reverts: #78829
Releases: master, 8.7
Reviewed-on: https://review.typo3.org/53014


Tested-by: TYPO3com <no-reply@typo3.com>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>