I second all the comments from TheoMD. I love Edgartown. The problem from a boat perspective is that the reserve moorings are a pretty good dingy run to town and for an expensive place, the facilities for boaters like showers, groceries, etc... stink. In addition, with the pond open to the sea, there was a pretty good current cooking thru the harbor. You can make mooring reservations on the website but they fill up fast. You could probably get a trasient mooring during the week without trouble as long as you arrive by around noon.
I have never stayed in Oak bluffs on a boat but would like to this summer. It is definitly the moust roudy of the towns. And the slips back right up to main street so I would not call it peaceful.
Vinyard Haven is much quieter with less to do. It is also a dry town if that matters.
Either way, you can't beat MV, it's beautiful and a great destination.