There are signs that voters are losing faith as problems mount for his administration ahead of midterm elections