I find this extremely hard to believe - privileged classes, rolling in dosh, not exploiting Africans when every other rich institution was?
From memory I think the original house was owned by Thomas Wentworth, the Earl of Strafford, who I think backed the winning royals until Charles the first used him as a scapegoat to appease Cromwell. I think the house passed to the Fitzwilliam family through a female Wentworth. I need to read up on it though.