I can’t say for your site, but i’ve worked on speeding up wordpress site’s before, and let me tell you, it’s not an easy task. Wordpresses template and plugin nature makes it prime for backend bloat. After optimizing our wordpress blog we get an average time to first byte of 900 ms (still not optimum by far). You can achieve lower times by doing the following:
1.) Install something like w3 total cache. Though i’d say do your testing with this enabled and disabled. I found that it actually slowed down the total load time for me since i had already done manual optimizations.
2.) Go through your template and look for anything being called dynamically that you can add in statically. This will reduce the general amount of queries needed for the template generation.
3.) Do a slow query log and see if there are any opportunities to optimize the database itself via indexes. If anything it will tell you if you have some badly written plugins running.
4.) Keep plugins to a minimum.
Keep in mind that some of this slowdown may be masked by database query and table caching. So the actual backend delay may not appear as bad for your most accessed pages.