Sky News understands Cabinet ministers have voiced concerns about the drive to cut EU migration in a number of low-income sectors after Brexit.
Downing Street is determined to end free movement and ministers have publicly referred to attracting "the brightest and the best" of highly-skilled workers in future.
But privately, some have argued that tens of thousands of low-skilled jobs currently filled by migrants cannot easily be replaced with British labour, if they face shortages.
Home Office officials have been holding informal discussions with Government departments calling for unskilled migrants to continue working in sectors including agriculture, hospitality, construction and social care.
One Cabinet minister said there is now an admission that "we can't just talk about high-skilled people; it has to be low skilled as well" or sectors of the economy would suffer.
"It's seasonal work like fruit picking which only lasts two months a year, but someone working at [the cafe chain] Pret could arguably be described as unskilled."
The issue was discussed at a meeting of the Brexit Cabinet Committee after a document drawn up by Home Office officials was circulated.