The rifled musket isn't my area of expertise..but, The usual suspects would be:
- Diameter of the bore/groove
- Diameter of the projectile
I assume the mini-balls are hollow-based...thus, the powder charge, granulation, etc must produce enough pressure to expand the hollow base to fit the bore. If more powder and/or finer grade do not help, look for a bullet with a thinner skirt or casts to a larger diameter (which may be harder to load). This should help it expand.
Fouling from shot to shot will also open groups.
Hope this helps,
Slim