Website only slow when going to certain domain name

My site loads in 1.063 seconds when I go directly to the IP address or our original domain name but takes 5.211 seconds to go to our current domain name.

Here is the webpagetest result for our site:

For some reason their is a much longer Wait when going to our domain vs going directly to our static IP address for the site.

Why is my site so slow? Could it be the DNS Lookup? How can I look at this configuration?

We have a domain name that the site was developed on but is not in use and this domain name loads quickly also and has 0 DNS lookups and TCP Connects.

So to clarify:

Example: my site, is slow to load but when you hit the IP address, (for example) it loads our site fast. When you hit the other domain name (which we don’t use), the site also loads very fast.

Why is my site slow to load? Any ideas?

It looks like the time is all back-end from WPT’s perspective. Do you have any vhost configs that might be doing something expensive (geo-location, reverse-DNS or something like that)?