Ok, 2 things.
When you remove material from the end of the arbor, you are indeed opening up the wedge slot. Number 2, make sure the wedge slot in the arbor goes further back than the rear of the slot in the barrel assy ( this can happen depending on how much material you remove).
To make the wedge you have work from now on, drill and tap the end of the arbor all the way into the wedge slot for a 1/4" X 20 by 3/8" long set screw. Smooth the end of the set screw. Now you have the ability to "adjust" the wedge slot as needed/ wanted for wedge width/position. Go slow and be careful not to break the drill bit when you reach the slot !!
Mike