Can surface owner be excluded from using building stone?

Where there is an “all minerals” or similar severance which includes building stone, does a surface owner ever need to be concerned about his right to use rock and small boulders located on the surface to build a structure on the property? Or does the close connection between such rights and the surface mean that a mineral owner’s rights to building stone will never exclude the surface owner from reasonable use of same?

My main concern is where all minerals are federal. I’m hoping the answer is a simple “A-OK, surface owner can always do that.” Citation to federal authority or related background reading would be much appreciated.