Both belt carry & haversack are feasible, so as far as authenticity, you could select either method. Personally I don't like excess weight on my belt as I'm usually toting two revolvers anyway...so, I favor keeping this gear in a shoulder-slung pouch. This also keeps my powder flask away from the firing line. I carry my caps in one or more cappers and always keep one on my person in event of a misfire.
Historically, my take is that the necessary flask, balls or conicals along with extra caps, cleaning equipment, etc. would likely have been kept in a pouch or sack which was generally stored at one's home, cabin or ranch.... or saddlebags or wagon when traveling afield. This would be during the period you requested (after 1865).
During the war, Northern soldiers exclusively used paper cartridges...it was the only form of ammunition issued. Confederates also issued paper cartridges, but due to their strained supplies of material, likely some loose ammunition was used in the field (user supplied). After the war, likely paper cartridges were available widely as surplus, but may not have stood up well due to age and rough handling. Hand cast bullets and loose powder loading would be most economical route then, just as today.
I've seen spare cylinders in only a few cased sets of firearms...& you'll find no period writing or discussions (only modern discussions). They did exist, but were certainly RARE.
Slim