UPDATE:
I noticed the format in the database is decimal(16, 2), I tried to change that to decimal(16, 3), but that did not take effect.
At this point I switched over to billing tax as a service template, calculated over the whole ticket, and somehow that seems to be holding it's total (so far).
As far as the customer is concerned, I obviously just print the {SERVICE TOTAL} on the line that has the text 'Tax:', and it is all the same to them.
Still would love to hear any ideas or suggestions as to why it is working the way it is...