I'm using HA + zigbee2mqtt for about 50 devices. I don't face your same problem at all. I remember I tried ZHA too at the beginning and it was not usable...