Since this is the History Forum, keep in mind that wearing anything Confederate that would indicate rank or display military buttons, such as the uniform blouse, was proscribed after the war.
Your pants, shirts, footwear, galluses, and perhaps vest, would be seen in use for a short time after the war until replaced by civilian clothing.
There are many examples of period uniforms in museums or as cherished family heirlooms, which indicates many soldiers kept their uniforms. Whether out of 19th century frugality, or pride of service, your choice.
Also, lots of veteran organizations, representing both Federal and Confederate, flourished in the decades post-war.
Many wore adaptations of service uniforms, and for the Confederates, probably of a better grade than worn in war time.
In the long run, this is a fantasy sport, so wear what pleases you.
Or, this would be a great reason to join GAF. A uniform is NEVER out of place. See The Barracks.
RCJ