Whether you're selling as a business or as a hobbyist, your time is still worth something. At ~$5 cost, that means you're paying yourself $15 per pen. I don't know how long it takes you, but it generally takes me about an hour for a single-barrel pen (assuming the blank isn't finicky, I don't make any mistakes, the stars are aligned, etc.). So $15 per hour. If you're comfortable with that, then I say great.
I do agree with others who have said you could sell those for more. I think it's important to find a balance between just covering labor and materials and charging whatever the market will bear. I would probably sell your pens for $45, but no more, even if I thought I could get more. As you said, the materials don't justify a higher price.
But take everything I say with a grain of salt. I've sold a number of pens online, but none in person. I'm doing my first show later this summer, so we'll see if people will actually pay what I think is fair.