After leaving office in 1981, Jimmy Carter faced significant financial difficulties primarily due to his peanut business, which was $1 million in debt. Despite these challenges, he did not have to sell his home. Carter and his wife, Rosalynn, continued to live in their home in Plains, Georgia.