First off let me say I wouldn't rebarrel an original Spencer. I might be convinced to reline/restore the rifling. There is a quality of relining that I've read about refered to as rifling restoration. It cost more then relining, and I've considered that for a 1876 Win, but I don't know the cost and details of it.
Your measurements: The bore Dia is .505 and the Grove Dia is .514. So your bullet should be .514-.515".
For comparison my 1865 Spencer is groove dia. of .520" and I use a Rapine mold 520350 that drops out of the mold at .520" and 350gr. I lube it with SPG or 50/50 NRA formula.Works fine.
I can't see how rough your bore is, but as your loads show the bullet has to fit the rifling and you saw a good improvement. I think if you could get the bullet weight down and the velocity up to 1000-1050fps that might improve things.
.