On-the-Job Problem Solving