I would guess it depends on what sort of speed you travel at. Most of my jobs are within a 4 mile radius, and maybe I wouldn't get above 30 MPH to get to them; so ladders or no ladders, I bet the difference they'd make to me is minimal.
But if your route involves a 40 mile return journey and you're doing 40 to 60 MPH, then ladders will have a greater effect on your fuel consumption.
I reckon.