I think the people who automatically classify Bayliners as "junk" are those who are new to boating, ignorant or uneducated.
It's funny, because all manufacturers had a few "bad" years. Look at Sea Ray, there were all those Balsa coring problems. The fit and finish is nice on the outside, but look at what you can't see. I grew up and learned a lot about boating on a Sea Ray center console. It looked fabulous on the outside, had nice lines, a good shine, just enough bright work, and I always kept it looking tip-top. However, there were stress cracks all over the place, soft spots in the deck, it was underpowered, wet, and would knock your fillings out in any kind of chop. Still, it didn't carry the same stigma that Bayliners do because it was a Sea Ray.
Pride of ownership goes a long way, too. Smaller Bayliners seem to have a reputation of being an entry-level boat, based on the MSRP and resale values. This may be a blanket generalization, but I would be willing to say that someone who doesn't spend much money on a boat may not take as much pride in ownership, which might lead to neglect and the conditions that are so common when people refer to Bayliners as junk. However, look at any other brand that has been neglected and you will find many of the same problems.
You get what you pay for, but you also get what you put into it in terms of labor and love. I don't particularly like the lines of older, smaller Bayliners. But I do really like the 38'+ range. If I remember correctly, there is an RO here who lives aboard a 38' Bayliner convertible, it has great lines and I have always admired that boat.
If you do buy a Bayliner, just peel off the manufacturer decals. Then the dock lovers and trash talkers won't make fun of you because they most likely won't even know the difference.