I do use filler in my cartridge revolver rounds, otherwise the folks at my home club sometimes claim that I'm damaging the targets, even though I am quite a bit below the max velocity. In cap 'n ball stuff, most of them will allow a ball or conical bullet seated down on a somewhat reduced load, so there is no need for filler until we get way low, say for a seven year old kid's first time.
Like Paladin Uk, I don't mind being frugal and I can scoop a whole lot of grits in while loading my .45 Colt to the old United States Cavalry powder charge of 28 or 30 grains of BP into a case. Since few people call the men of the US Cavalry wussies, except of course US Infantrymen, I feel quite comfortable loading those smaller charges. After all, Saint John of Browning used the 28grain BP behind a 230 grain bullet ballistics for the final version of the 1911 pistol load.
I do not wish to start fights here, I just note that there are valid reasons to wish to tone down the velocity or, Perhaps, the muzzle report of our loads, hence filler can be valuable. Sometimes.