iirc you can use the key to open the cave passage that leads to the bandit's hideout, you can use it at the gate going south from the wizard's camp
