There are lots of things that in the ordinary world of drilling that don't work very well in drilling blanks for pens. In a 2x4 board, into a sheet of plastic, into metal, there are specialty bits.
But in pen blanks, you don't want heat build up. The blank is small, narrow, no bulk around it like in a 2x4 board or in a large sheet of plastic. The thin blank will be affected by heat, both plastic resin and wood. The thinness of the blank means less bulk around it to keep it intact, therefore "blowouts" and cracking occur when it wouldn't in normal bulky medium drilling circumstances. The type of bit used in bulky situations where heat is not too much of a problem can devastate a blank.
Ergo, what is common in the normal world of drilling is not what necessarily works in small blanks. The fellows above gave some good advice.
Drilling straight: If you can, use the lathe as mentioned. Are you using a drill press? And do you have a pen vice? GET a good one if you don't, Clamp down the pen vice on the press; don't let it move. Take small bites, in and out, in and out, in and out. Even on the lathe, take small bites and remove, small bites and remove. The bits do not eject the material at a very good rate, so that is the reason for small bites and remove. a build up of material will get hot and cause all kinds of trouble.