Even the prime minister's most loyal ministers are pushing him to change, writes Laura Kuenssberg.