I don't have a Colt SAA, but I've had Remington NMA's in both 8" and 5.5" barrels, so I can tell you how the same gun differed with different barrels.
As Deadeye said, it's really a personal preference, but for me, I love the shorter barrels, hands down.
For CAS, it was simply too hard to draw those long shootin' irons. I really had to bend the wrists to uncomfortable angles to clear the holsters. The shorter guns are much, much easier to draw.
And as Deadeye said, I just prefer the balance of the shorter guns. But, I also agree than the shorter barrels are harder to aim. I do have a harder time "finding the sights" with my shorter guns, but I feel their advantages far outweigh this downside.
However, given all this - I wouldn't go any shorter. I feel 5.5" inches is a real sweet spot for me and my revolvers.
If I had it to do all over again, I would definitely have tried to take as many guns for 'test drives' as possible before I bought. Picking a gun for looks alone will just lead to regrets down the road.