I think what happened here was that Her Interactive wanted to do a game that took place in the Old South and include a big spooky mansion and Spanish moss and all that stuff -- and then found themselves face to face with the real nightmare of slavery. They didn't want to pretend like it didn't matter or gloss over it, and if they dealt with it realistically it would be too horrible for a lighthearted computer game. So they kind of ... sidestepped it, letting you, the player, fill in the details for yourself.
My guess? Based on the cotton gin, the family ran a cotton plantation and mill, and those 54 souls were slaves. Or were they? You decide.