Tehran said vessels violated protocols for travelling across waterway and could face fines or confiscation