Yea I sand with the metal bushings still on.
How should I sand instead? With non stick plastic bushings?
That is your problem. An absolute easy way which very few seem to grasp is to use a very sharp and just honed scraper at 3000 - 3500 rpm.
A few use a skew, but I cannot speak to that as I am not proficient with a skew.
I am not touting this pen which I did 12-13 years ago - but look at the metal, and particularly the dots. The dots are solder and smear like pencil lead - easy. Notice those dots, not a bit of smear, and they are much softer than the metal bushings, and smear much worse than bushings. Yet, I did not get any smear, just clean and clear cuts.
This is a view of the clip end of metal and wood segments from a pen made in late '08 and used almost daily since then. Do segments move, shrink, grow? Here is a photo of my one of my pens and experience.
www.penturners.org
Elegant Beauty TI Gold Bloodwood, BOW, Holly core solder CA finish
www.penturners.org
What I think most people do not grasp is that sandpaper (SP) smears and makes very very fine dust. Duh, we all know that, but SP creates dust and moves different colors (metal or wood) onto other wood. There is a simple way to stop that - I use a flat end scraper that has been rounded in the same manner as a radiused square carbide insert, only my scraper is an HSS. The scraper will scrape rather than sand, and all kinds of segmented woods and or metal stay as clean as they can be without the sanding dust. AND if one takes the time to do pristine sharpening, it will be as smooth as a baby's behind (an American colloquialism) with out the need for sanding.
Apologies for jumping to the main solution and bypassing a step: Plastic bushings will help, but the best way is without bushings and without a mandrel. TBC - Turning between Centers - Bushings are used between centers for most people to get it close to size, and then take the bushings off and place the blank directly between the Centers. then finish to size and finish. This method does allow for sanding if you want without getting sanding dust on the blank. . . . Still the scraper method works too. A few people use a skew instead of a scraper but lay the skew on its side on the tool rest and in effect use it like a scraper.