At least I got it stable now by backporting Nvidia drivers 570;

Still slowly looking for the next replacement card that is not going to be Nvidia.