A group representing the bereaved of Grenfell call Simon Dudley's comments "ignorant and callous".