Making the Most Out of Supervision