Maine. Everything else pales.
But, seriously, check out the Freedom Trail in Boston. I could spend weeks just hitting everything along the trail, and when I went, we only spent about 4 hours. With an 18-month-old kid. Yeah, I missed a bunch.
Plymouth is ok, but nothing spectacular. Didn't spend enough time on the Cape, but it was gorgeous.
Newport=$$. Nice town, great views of the oceans and the houses.
Ditto the "meh" on Mystic Pizza, but it's a cool town.
But, by all means, go to Maine. We had a weeklong trip planned to southern New England. Boston, the Cape, Newport, and Mystic. We spent 3 days in Boston. After eating at the famed "Legal Seafood" and being disappointed, we hopped in the car the next evening, headed for Maine. Having been there (Bar Harbor) on our honeymoon, we knew what to look for.
We got off I-95 in the first exit over the border from NH, (Kittery? I think it had a bunch of outlet shops, a plus for the wife) and relied on the GPS to get us near the water. We found a little hole-in-the-wall seafood joint, that absolutely blew Legal away. Wonderful lobster, chowder, everything. Name? Sadly, that was 3 years ago. . .memory is failing me.