Passing the Baton: Learning to Listen