Why is Databricks no longer recommending the use of mounting external data locations to Databricks Filesystem and using credential passthrough with S3?
Databricks no longer recommends mounting external data locations to Databricks Filesystem and using credential passthrough with S3 due to deprecated storage patterns and potential issues with resource management and directory marker handling. Specifically, the S3A filesystem enables caching by default and releases resources on 'FileSystem.close()', which can cause issues if other threads are using a reference to the cached file system. Additionally, the S3A filesystem does not remove directory markers when closing an output stream, which can cause problems for legacy applications that misinterpret them as empty directories.