Read CNN Business’ last interview with Madoff

Editor’s note: This story was originally published on May 16, 2013 and has been updated with Madoff’s death.

Bernard Madoff once had billions of dollars, but in the last years of his life, he made $40 a month doing menial prison labor.

The mastermind behind the largest Ponzi scheme in US history died at 82 on Wednesday.

“I used to work as a clerk in the commissary, and now they have me taking care of the telephone and the computer systems,” said Madoff, speaking by phone from a federal prison in North Carolina. His prison phone account didn’t have any money in it, so he had to call CNN Business collect.

As far as the computers and phones go, Madoff said he has to “make sure they’re working and they’re kept clean,” but he emphasizes that this requires no technical skill whatsoever.

It’s a far cry from his former life. Once head of his own finance firm, Madoff is now identified as inmate #61727-054 at Butner Federal Correctional Complex. He’s serving a 150-year sentence after pleading guilty, in 2009, to siphoning $17.5 billion from thousands of investors, wiping them out in a long-running, pyramid-style deception.

He said he only works “a few hours” a day, which leaves him plenty of time to ponder his undoing.

“I’m usually up at 4:30 in the morning because I can t sleep,” said Madoff, 75, who is scheduled to be released in 2139.

Certainly his surroundings in a medium-security prison are a lot less comfortable. When his massive Ponzi scheme unraveled in 2008, Madoff had to give up his $7 million Manhattan penthouse, a beach house in Montauk, N.Y, his homes in Florida, and France, as well as a yacht named “The Bull.”

Madoff took great pains to insulate his family from his financial wrongdoings, but it didn’t work.

He’s especially haunted by the 2010 suicide of his oldest son, Mark, who hanged himself on the second anniversary of his father’s arrest.

“I was responsible for my son Mark’s death and that’s very, very difficult,” he said. “I live with that. I live with the remorse, the pain I caused everybody, certainly my family, and the victims.”

His younger brother Peter, 67, is serving a 10-year sentence at a medium-security federal prison in South Carolina, after pleading guilty to covering up the financial wrong-doing.

“Obviously, the main concern that I have is being away from my family,” he said. “Married for 50 years, I had a very close family.”

Madoff said that his deception began after the Black Monday crash of 1987, a massive stock market sell-off from which he never truly recovered. He said that’s when he turned his investors into victims, but he insists that his crime was never supposed to go on for so long.

“It was certainly never my intention for this to happen,” said Madoff. “I thought I could work myself out of a temporary situation but it kept getting worse and worse and I didn’t have the courage to admit what I had done. It created a major problem.”

Many victims of Madoff’s “problem” had hoped he would suffer in prison. But Mike De Vita, a Madoff victim who co-authored “The Club that No One Wanted to Join,” observed that “life for him is kind of good in some ways.”

De Vita doesn’t believe that Madoff feels any true remorse for the damage he caused to others, even to his own family. He dismissed Madoff’s remorse as “words, and words alone.”

“How could a father bring his own two sons into a business that he knows is nothing more than a massive criminal enterprise?” De Vita asked. “If he has that little consideration for his own family, how much consideration do you think he has for us?”