According to the Addressables ⁠documentation the reported issue of missing reference seems to be an intended behavior, it is working nature of Addressables to optimize the unused assets in a scene.
Please see https://docs.unity3d.com/Packages/com.unity.addressables@2.3/manual/UnloadingAddressableAssets.html

* When we are loading even the addressable scene using Addressables.LoadAsync() with default parameter of LoadSceneMode.Single the Unity runtime unloads the current Scene and calls Resource.UnloadUnusedAssets which unloads assets that are not used.

* Also, the methods of preserving the assets such as object.DontDestroyOnLoad or HideFlags.DontloadUnusedAsset do not work.

Unused Assets in Addressables:
* An asset is deemed to be unused if it isn't reached after walking the whole game object hierarchy, including script components. Static variables are also examined.

* However, script execution stack is not examined so an asset referenced only from within the script stack will be unloaded. but all the assets other than ScriptableObjects are loaded back in the next time one of its properties or methods is used.

Workaround as per Addressable Documentation:

* To prevent the scene and its assets from being unloaded, keep a reference to the scene load operation handle until you want to unload the scene manually.

* To do this, use ResourceManager.Acquire which Increments reference count of operation handle.

Example Code:

private void OnInitCompleted(AsyncOperationHandle<IResourceLocator> obj)
{
var handle = Addressables.LoadSceneAsync(m_SceneKey, LoadSceneMode.Single, true);
resourceManager.Acquire(handle);
}