We can implement referrals support for Kerberos mk_req API or do it for GSS-API. For me its an easy choice, most consumers uses GSS-API, lets go for that.
DNS canonicalition happens when we pass in a GSS_C_NT_HOSTBASED_SERVICE based name into gss_import_name(), so its only that case we really need to care about. DNS canonlisation is done by krb5_sname_to_principal(), see documentation in Heimdal trunk.
The simple solution is that in gss_import_name() we the hostname to krb5_sname_to_principal(), instead we mark up the principal as magic-unresolved. Now there are some places where we should return the name in its native form, and that is after gss_init_sec_context() is done by gss_inquire_context(), and gss_canonicalize_name(). We also need to deal with these magic-unresolved names in gss_acquire_cred() and gss_export_name().
So lets go over the four cases when we have a magic-unresolved.
gss_init_sec_context(), we try first to get Kerberos credentials using TGS-REQ using the sort name and hope that its an valid alias, if no, we call back to the old way with krb5_sname_to_principal(). Right, we update the name in the context to what we used.
gss_acquire_cred() same thing here, first try literal, then krb5_sname_to_principal().
gss_inquire_context(), this is simple gss_init_sec_context() will have solve the problem for us, just copy out the string.
gss_canonicalize_name() here we don't have a chance, so I guess, we want to have the canonicalized name, so I used the krb5_sname_to_principal() to canonicalize the name.
So where to store this magic-unresolved bit ? Well it happens to be so that the Kerberos gss-api mech uses Kerberos principals (krb5_principal) so there no extra bit where we can save the info. So, instead we hook the krb5_principal name type field and misuse that. Any time we will export or use the principal, we'll convert back to KRB5_NT_SRV_HST that it would have been if we used krb5_sname_to_principal().