If you have lost him, go back to the store and convince him to go to the Sword Master again. Remember, in order to be able to follow the storekeeper out of his store, you can't be holding any of the store's items you haven't yet bought; he'll accuse you of shop-lifting if you are.