What brand of collet chuck? What thread are you adapting from/to?
Reason I ask: I have 3 PSI collet checks and they have all performed great with an expanding mandrel. I also have the Beall collet chuck and it runs fine with a mandrel. The PSI chucks come with an adapter in the box to be used depending on your lathe nose thread.
With the PSI collet chucks you have to install the collet into the knurled nut, first, before you assemble the chuck or the mandrel will not run true!
Just snap the collet into the nut, I place the nut face down on the table and snap the collet in place, then you screw the collet/nut assembly into the chuck body, insert the mandrel and finish tightening the chuck.
If you don't follow the steps, exactly, the mandrel will not run true.
Fred