[BUGFIX] Prevent PHP type error with deprecated settings for mailer

This patch converts the $mailSettings['transport_smtp_encrypt'] explicit
to a bool as expected by Symfony/Mailer.

Additionally this setting is corrected in the admin tool which was not
adapted correctly with the introduction of the Symfony/Mailer.

Resolves: #90295
Releases: master
Change-Id: Ia5f48eec810bfda7936d462c098f5d5eba4332c0
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/63146


Tested-by: TYPO3com <noreply@typo3.com>
Tested-by: Martin Hotmann <martinhotmann@gmail.com>
Tested-by: Benni Mack <benni@typo3.org>
Reviewed-by: Oliver Klee <typo3-coding@oliverklee.de>
Reviewed-by: Andreas Fernandez <a.fernandez@scripting-base.de>
Reviewed-by: Martin Hotmann <martinhotmann@gmail.com>
Reviewed-by: Benni Mack <benni@typo3.org>