In Palm Beach and Martin counties, a number of people who knew Matthew Brown regarded him as a caring, generous serial entrepreneur who would drop everything to help those in need. He sponsored Pop Warner football teams, supported Gold Star military families and aided people with medical problems.
But for years, the IRS and federal prosecutors say, he deceived the clients of his payroll service firm, diverting millions that were supposed to be paid to Uncle Sam in taxes. The money, the government says, went to buy more than two dozen Ferraris among other exotic cars, a jet plane, a yacht and various pieces of residential and commercial real estate.
Brown, 51, of Palm Beach Gardens was sentenced to 50 months in prison this week by U.S. District Judge Aileen Cannon. In January, he pleaded guilty to not paying employment taxes withheld from his employees’ pay, and to filing a false tax return.
In a statement Thursday, the Justice Department said that between 2014 and 2022, Brown failed to pay more $20 million in taxes withheld from the wages of employees of clients of his firm, Elite Payroll, “and from other businesses he controlled and instead enriched himself.”
“To effectuate his scheme, Brown charged his clients the full amount of their tax liabilities but then filed false employment tax returns with the IRS that substantially underreported their liabilities, and pocketing the difference,” the government said. “For example, for one quarter in 2021, a client owed approximately $219,000 in taxes. Elite Payroll collected that amount from the client but filed a false tax return with the IRS claiming that the client only owed approximately $32,000, which Elite paid. Brown then kept the remaining approximately $190,000.”
Over the years, the government said, Brown bought commercial and residential real estate including his multimillion dollar home, as well as the following:
- Up to 48 vehicles including a collection of 27 Ferraris, a Porsche, and Rolls Royce.
- A high performance Valhalla 55 Sport luxury cruising yacht.
- A Dassault Falcon 50 long-range business jet.
But a number of individuals including businesspeople, lawyers and employees of his firm wrote character references to Judge Cannon asking that she be lenient when sentencing Brown.
Divergent stories
In contrasting presentencing filings, the government and defense disagree on how Brown’s tax crimes unfolded.
Andrew Ascencio of the Justice Department’s tax division argued that Brown diverted customer funds — money that should have gone to the IRS — to his own use.
But defense attorney Jeffrey Neiman asserted that was not the case.
“Against the backdrop of his difficult upbringing and family life, Mr. Brown made the worst decision of his life to violate the law,” Neiman wrote the court. “While his conduct is unquestionably serious, some context is required. Mr. Brown’s introduction to the payroll and human resource services industry came from his father. After graduating from college, Mr. Brown’s father — who had kicked his son out of the house only a few short years before — asked Mr. Brown to move from Rhode Island to South Florida to work for him at what would later become Elite Payroll Solutions.
“While working for his father’s company, Mr. Brown learned that the business operated at a low profit margin and therefore routinely used funds from one customer to pay the past-due payroll taxes for another,” Neiman said. “Initially, Mr. Brown was taught that this was simply how business was run and that there was nothing inherently unlawful about doing so. By the time he took over the reins of the company from his father, however, Mr. Brown learned otherwise. Though we are by no means minimizing Mr. Brown’s actions, it is critical to recognize that he didn’t start the practice.
“He simply did what his father taught him to do. While he should have had the courage to put a stop to the unlawful practices, he knew that doing so would expose his father’s misdeeds. Despite [or, more accurately, because of] their complicated history, Mr. Brown’s misguided loyalty to his father unfortunately won out.”
Neimen went on to say it “is simply not true” that his client “used customer funds slated for the IRS to enrich himself and fuel a lavish lifestyle.”
“Neither Mr. Brown nor any of his companies kept any monies paid by customers of Elite,” he wrote. “Rather, consistent with how he was taught, Mr. Brown used most of those funds to pay other customers’ payroll taxes for prior years, with the remaining funds used to keep his father’s other failing entities afloat.”
His client currently owns “in whole or in part,” more than two dozen businesses that were not involved with his offenses “and are otherwise pristine,” Neiman wrote
They employ 150 people in Martin County alone, he added, the largest of which is a Stuart company called Tee Off, a golf course, as well as a nationwide landscape staffing company with 4,800 employees nationwide that generates $210 million in annual revenue.
The government didn’t buy it, noting that Brown used deception to throw off the IRS when its agents started to investigate irregularities at the payroll firm.
Ascencio wrote that Brown blamed the underreporting of tax obligations on his own company’s systems or employees, “and immediately offered to pay any balance and interest due for the flagged period.”
“The Defendant did this to avoid further scrutiny of Elite Payroll by either the IRS or his clients,” Ascencio wrote.
One of those clients, a dentist whose name was redacted from an affidavit filed by the government, said he spent $200,000 to hire an attorney to help him clear his own name after IRS agents knocked on his door in 2023 and presented him with a summons. The dentist said he retained a forensic accountant to determine whether Brown had stolen funds from his business account.
“I trusted Matthew Brown to remit the tax withholdings to the IRS and he willfully failed to make those payments,” the dentist wrote.
A nine-figure net worth
Ascencio didn’t buy the theme that Brown got rich through honest dealings.
“The Defendant prospered from the funds stolen from his clients,” the prosecutor asserted. “As stipulated by the Defendant, ‘instead of paying over the funds he held in trust for his clients, Brown purchased commercial and residential real estate including his multi-million dollar home and a collection of high end luxury cars.’”
He wrote that Brown, “through his fraud and deception … now has a nine-figure net worth.”
The prosecutor wrote Brown acquired “as many as 48 vehicles including a Porsche, Rolls-Royce, McLaren, Land Rover, a collection of 27 Ferraris, three boats, and a plane.” He made real estate purchases “through at least 18 real estate holding companies.”
Ascencio also noted that Brown owns “at least eleven businesses including a staffing company, HVAC business, and an automotive shop — among many others.”
To date, Ascencio said, “the government has identified $24,813,167 of unpaid employment taxes associated with Elite Payroll and seventeen other entities that had returns prepared by Elite Payroll.”
Besides the prison time, Judge Cannon ordered Brown to pay more than $22.4 million in restitution and a $200,000 fine to the U.S. Government. Brown is also to serve two years of supervised release after his incarceration. He was remanded into the custody of the U.S. Marshals Service.
The IRS criminal investigation division developed the case. Ascencio and Assistant U.S. Attorney Michael Porter in Fort Pierce were the prosecutors.