Retired neurosurgeon Ben Carson has told President-elect Donald Trump that he isn't interested in serving as secretary of Health and Human Services, a Carson ally confirmed to The Hill on Tuesday.

Business manager and close friend Armstrong Williams said Carson won't join the incoming Trump administration and would only serve as an unofficial adviser. 

Circa on Tuesday reported that Carson had been offered the position, citing Williams. But Williams told The Hill that no specific offer had been made.

"Dr. Carson was never offered a specific position, but everything was open to him," Williams told The Hill in a phone call.

"Dr. Carson feels he has no government experience, he's never run a federal agency. The last thing he would want to do was take a position that could cripple the presidency."

The renowned neurosurgeon was one of Trump's first primary opponents to jump on board his campaign. He was viewed as one of the likely picks to lead the agency that will work to roll back ObamaCare in the new administration.

Reports also pegged him as a candidate for secretary of Education.

Earlier this month, Trump described Carson as a "brilliant physician" and said he hoped Carson "will be very much involved with my administration."But Williams said Carson would only advise the president-elect unofficially.

Carson has been looking for ways to be influential outside of government, including his My Faith Votes, a group that looks to motivate Christian voters.