Reproduction steps:

# Open the user’s attached project “UnityTestProject“
# Go to Window > Package Manager and install Android Logcat
# Open "/Assets/SampleScene.unity" Scene
# Rename the dll files in “MinimalLib > Android > armeaby-v7a” to XXX.so instead of libXXX.so
# Go to Build Settings > Android and tick Development Build
# Build And Run
# Untick Development Build
# Build And Run
# Observe Android LogCat

Expected result: The error "DllNotFoundException: Unable to load DLL ‘Minimal'“ is not found in Android LogCat
Actual result: The error "DllNotFoundException: Unable to load DLL ‘Minimal'“ is found in Android LogCat

Reproducible with: 2020.3.42f1, 2021.3.14f1, 2022.1.23f1, 2022.2.0b15, 2023.1.0a17

Reproducible with devices:
VLNQA00332, Samsung Galaxy XCover4 (SM-G390F), Android 9, CPU: Exynos 7 Quad 7570, GPU: Mali-T720
VLNQA00099, Google Pixel XL (Pixel XL), Android 10, CPU: Snapdragon 821 MSM8996 Pro, GPU: Adreno (TM) 530
VLNQA00325, Samsung Galaxy Note10 (SM-N970F), Android 12, CPU: NOT FOUND, GPU: Mali-G76
VLNQA00318, Oneplus OnePlus 7 Pro (GM1913), Android 11, CPU: Snapdragon 855 SM8150, GPU: Adreno (TM) 640
VLNQA00122 - Samsung Galaxy S9 (SM-G960F), CPU: Exynos 9 Series 9810, GPU: Mali-G72, OS: 10.0.0

Reproduced on: macOS 12.4 (Intel)