Have you talked to Herman Toothrot?
He says the cannibals had the key at one time...
But now he has the key to the Monkey Head.
He refuses to give it back to the cannibals until they give him back his banana picker.
If you give him back his banana picker, he'll give you the key to the Monkey Head.
How can I get the Banana Picker?
After you get the banana picker, you can give it to Herman at the cannibals' village or, if he's left the village, in his fort.