I figured out the same thing on my own, but the video posted by
@its_virgil above is better than what I was doing. I was using small diamond files and moving them back and forth on the flat faces. That made it harder to keep the diamond file properly registered against the flat face.
The video shows how to place a diamond card extended slightly over a workbench edge, register the flat face of the pen mill, and then move the mill back and forth on the stationary card. (Repeat for each face.) That's a better approach.
Quite honestly, I got fed up with sharpening those cheap pen mills I was buying from Rockler, which dulled quickly, and was too lazy to make my own disc sander pen milling jig on the lathe, which would have been better. I sucked it up and bought the Woodpeckers pen mill. I don't like to buy Woodpeckers tools because I think they are way overpriced, but after shopping and comparing with similar carbide pen mills, I bought the five piece set, which includes pilot reamers that fit the pen kits I make. It is only the second Woodpeckers tool I've bought. I can vouch for the high quality and ease of use of the Woodpecker pen mill. It can be aggressive if you are not careful; use a light touch.
What I like about carbide pen mills is the ease of sharpening. You rotate the carbide tip. If all edges have been used up, then you can try flattening the back on diamond stones or just buy replacements from Woodpeckers or a third-party supplier like Arizona Carbide.
Woodpeckers pen mill:
https://www.woodpeck.com/ultra-shear-pen-mill-inserts.html
Arizona Carbide, one of the vendors who is active here on Penturners and a great resource for inexpensive, but good quality carbide replacement tips. They are a far better value than the original tool maker:
https://azcarbide.com/
https://www.penturners.org/forums/azcarbide.410/
(Don't ask me which replacement to buy, I have not needed to replace them yet.)
Edit, Addendum:
Rockler sells two kinds of pen mills. One comes in a set with "barrel cleaners" in common pen kit sizes and a six edge mill. I have two of those. When one got dull, I could use the other until I had a chance to sharpen the dull one. Because they have six edges, the spacing is tighter between them for sharpening.
The other pen mill at Rockler was sold as separate parts - you bought the mill and the barrel cleaner separately. I just looked, and Rockler's website has the four-edge mills, but only a 7mm barrel cleaner on the website, leading me to wonder whether they are phasing it out. Those four-edge mills are similar to the one sold by Woodcraft in the thread title.
Rockler six-edge pen mill kit, I made pens with it, but got tired of the tricky, frequent sharpening and they are expensive to replace:
https://www.rockler.com/barrel-cleaner-pen-mill-kit
Rockler four-edge pen mill kit that I didn't buy because the overall cost was higher than the six-edge pen mill kit above, and now it only seems to have a 7mm barrel cleaner only - a two part kit, or the separate parts:
https://www.rockler.com/7mm-x-1-2-pen-pencil-combo-mill
https://www.rockler.com/pen-mill-barrel-trimmers
https://www.rockler.com/7mm-pen-barrel-cleaner
Woodcraft's Pen Mill:
https://www.woodcraft.com/search?q=pen+mill
They have many reamers (barrel cleaners). Use this search to see them:
https://www.woodcraft.com/products/whiteside-pen-mill-set
