See, people like my mom would insist some weaksauce statement like "oh but they shouldn't get physical".

It's self defense. No contest. Nazis are an inherent threat to everyone, like any other fascist.