The oysterman maintains a strong base of support, but his scandals have left a bad taste with some Democratic and independent voters.