There was a whole economy based on small farmers and businessmen leasing slave labor from slave holders.
"They too needed emancipation"
US Grant