A good place to ask this question (and get an accurate answer) would probably be over on the NSSA's website forum.
www.n-ssa.orgThose pards over there shoot their C.W. rifles and carbines in target matches and I'm sure some of them could give you some good advice on the subject.
That said, I know that there are some special bullet moulds made up that are available that vary by .001" increments when you order them. This way you can custom "tune" you bullet's fit to your particular rifle's bore diameter for best accuracy.
Another interesting aside is to look up the "Volunteer Rifle", which was a British Target Rifle of the same era.
The main Rifle-Musket of the American Civil War on the CSA side (pardon me, War of Northern Aggression) was the 1853 Enfield.
Many Federal troops were also equipped with the same rifle.
Over in England there were the "Volunteers" (their 1860s version of our National Guard).
The Volunteers for their drills took up target shooting matches as one of their main pastimes.
For long range shooting their standard issue Enfields really weren't up to the task so they adopted rules to allow custom target rifles to be built by private gunsmiths based on their Enfields as a basic pattern. (The replicas look essentially like a 3-Band Enfield but have better sights)
These rifles ended up being slug guns (not Rifle-Muskets) shooting long solid Lead bullets of .45 caliber. (These rifles essentially became caseless ".45-60s")
The closer tolerances made for slower loading and more frequent cleaning than the Rifle-Muskets, but accuracy was much better at longer ranges.
Replicas of these Volunteer Rifles have recently been made and (I think) are still available.
You might consider one of these in your search for an accurate "military" M.L. rifle?