What did the Three Important-Looking Pirates say about finding the Sword Master?
"Someone in town can probably direct you."
In fact, someone can. But not directly.
Have you visited the far side of town?
Have you talked to all the people there?
The storekeeper, in particular, could prove helpful.
Tell him you're looking for the Sword Master.
He'll volunteer to go over there, leaving you in the store.
And he walks pretty slowly.
That makes it easy for you to...
Exit the store, and walk after the Storekeeper. Follow him. He'll lead you to the fork in the road. (Labeled "Fork" on the map of Melee Island, directly above the "Lookout".) Once in the woods, continue to follow him. He'll lead you to the Sword Master.
If you have lost him, go back to the store and convince him to go to the Sword Master again. Remember, in order to be able to follow the storekeeper out of his store, you can't be holding any of the store's items you haven't yet bought; he'll accuse you of shop-lifting if you are.
The Sword Master's home now appears as the yellow dot on the hill (labeled "Sword Master's") above and to the left of the "Fork." You can now walk directly there, avoiding the storekeeper shenanigans.