Sky News has learnt that lawyers at RBS are in advanced talks with the Federal Housing Finance Agency (FHFA) about a deal that is almost certain to cost the state-backed bank more than $4.5bn (£3.5bn).
A source in Whitehall said that the discussions had progressed sufficiently far to leave both sides hopeful that an announcement can be made in the next few weeks.
The precise size and timing of the fine are still moving around and are subject to further negotiations, the sources said, with more detailed discussions said to have taken place in recent weeks.
One insider said there were concerns in Whitehall that the penalty could significantly exceed $4.5bn.
The settlement with the FHFA relates to the mis-selling of mortgages to the US government-backed loan firms Fannie Mae and Freddie Mac prior to the 2008 financial crisis, when RBS was among the biggest players on Wall Street.
RBS executives are keen to agree a settlement as soon as possible as they continue their efforts to return the bank - which is more than 70%-owned by British taxpayers - to profit for the first time since 2007.
However, they also have to engage in formal settlement talks with the US Department of Justice about another big penalty related to residential mortgage-backed securities - a process which has been delayed by a clearout of senior officials at the agency under President Donald Trump's new administration.
The DoJ fine will cost billions of pounds more.
Nevertheless, a deal with the FHFA would be another important step along the road to helping RBS shed its remaining financial liabilities from the pre-crisis era.
Philip Hammond is scheduled to give the Chancellor's annual Mansion House address to an audience of bankers and business leaders on Thursday, and he may signal his intentions on the future of the taxpayer's stake in RBS.
Mr Hammond, whose focus in his speech is likely to be on the impending Brexit negotiations and their impact on the City, has been clear that he does not believe the Government should be a long-term shareholder in RBS.
Before the General Election, Mr Hammond said that fair value for the stake could be below the 502p paid by the Labour government in 2008.
RBS shares now languish at 250.1p, just under half the average level at which £45.5bn of public money was injected to save it from collapse.
If the penalty from the FHFA is in the region of $4.5bn, or slightly more, RBS will be able to draw on existing provisions it has set aside to pay fines for RMBS mis-selling.
In January, RBS took a further $3.8bn charge for settling regulators' investigations, taking the total allocated - and so far unutilised - to £6.7bn ($8.3bn).
It is likely to have to add to that if the eventual DoJ penalty is at the higher end of analysts' forecasts.
RBS is the last of 18 banks to settle with the FHFA, although a number of other banks are also yet to settle with the DoJ, including Barclays, which is embroiled in a legal battle with the agency.
The mis-selling of residential mortgage-backed securities (RMBS) has been - in terms of the scale of the penalties - the biggest scandal to emerge from the pre-crisis era, with investors duped into buying toxic mortgages that were packaged up by banks and sold as ultra-safe.