I agree with whats been said. Captains cove is a great place to take kids on a sunday afternoon. They have live miusic outside form 4 to 7. I don't think it's a good over night place with kids, although, I've never stayed over.
I'll second norwalk as a good destination. The aquarium is great for the kids and adults. They also have an Imax movie. The town is a short walk from the town docks and has good food and shops.
How long do you plan on going for?