I would imagine the high speed on the lathe (at least with the 3/4" bit - if not the 1/2" bit as well) is causing issues. First - it creating smaller chips that will tend to pack rather than flow out the flutes (causing heat and affecting hole size. Secondly - the small chips get dragged around with the bit or in the way and your not only rubbing chips on the bore (creating heat) but also re-cutting chips (also creating heat and making even smaller chips). Thirdly, for a good chip load, at high RPM, the feed of the drill has to be proportionately faster... and wood doesnt handle that well without a very rigid setup (both blank and drill - which a wood lathe with a 3/4" bit - is not). And with the increased speed and disproportionately slow speed also causes the cutting edge to rub more rather than cut, dulling it.
Accurate and clean hole drilling is one of the more difficult parts of call making. Oh so often overlooked and underrated. You have to learn your lathe, drills, and technique for drilling well with your setup - just like you have to with turning, finishing, tone boards, etc... I know some drill at insanely high speeds compared to what I think is best, but I dont have their equipment either.
I would suggest, 500-700 max RPM with a 1/2" bit and a some what aggressive feed - and peck drilling with full retract to clear the chips often - more often with some woods, and less with others (let the flute packing be your guide). With a 3/4" bore... I would go about half that RPM. If it were me, any drill at 1/2" or larger - I would always use the slowest speed the lathe has. But everyone has their own style.
That is one issue with a lot of the wood lathes I see... so few of them really go slow enough to make life easier for call makers drilling barrels. 1000-1500 rpm isnt a big issue with a 1/4" drill... but get up there in diameter and it causes drama. Especially since wood lathes are not inherently accurate or rigid in terms of tail stock to spindle alignment and spindle and quill size. I wish wood lathe makers would put an extra pulley groove, that would get in the 200 rpm range... that would help a lot of people out, and help make larger drills live longer.
Chucking reamers are another way to get a better surface finish on the bore, as well as a much more accurate bore (spoiler alert - just because a drill bit is a certain size - it does NOT mean thats what size hole you will get). But the down side of using a chucking reamer is its two operations, drill undersize by 1/32 or 1/64" and then run the reamer through. And another issue is again RPM... reamers are designed to run at about 1/2 the speed of the same size drill, and fed into the material twice as fast. And I dont know of a standard bench top type lathe that will go down to 75-100 RPM or a quill that allow you to feed the reamer in fast enough to keep things going without having to be as fast as The Flash. But you can circumvent that some by locking the quill and sliding the whole tail stock body. That is made possible because chucking reamers follow rather than go out on their own... so the alignment of the tail stock to spindle is less important with a reamer.
If the diameter of the entrance of your hole is larger than the exit, there is an issue... either with the grind of the drill or the alignment of the bit to the spindle centerline. If the hole is the same size on both ends, youre in good shape. But thats a whole different tangent :D
Hope that helps!
Wade