Mushrooms are easy to prepare badly (usually undercooked), which really doesn't help their rep.

It also doesn't help that the tastiest mushrooms are all horrifically expensive, so most people are eating various growth stages of agaricus bisporus, which mostly just tastes earthy and not much else.