The Tories have accused Labour of "running scared" after ministers denied them a chance to delay Parliament's summer break.