So, I picked up a round-tipped mini carbide tool from JAustin, who's a member on THO, and the call I turned last night with an aluminum inlay cut like butter. No doubt the sharpness of my HSS tools was the issue. Eventually, I need to pick up a Wolverine jig to aid in sharpening my HSS tools. But, this was a cheaper option for now and I'm very pleased with the cutter from JAustin.
Thanks for all the input and wisdom from my THO brothers.