Big gap between Initial Connection and TTFB

Web Performance Discuss Test Results
1

I’ve always been into optimizing page load time. I joined Webpagetest to learn more on optimizing techniques. Anyways, here is my test result:

http://www.webpagetest.org/result/111207_Z5_2E1NA/1/details/

HTTP server: lighttpd/1.4.28
PHP version: 5.2.17
XCache Enabled
MySQL version: 5.1.39-log
Application: Invision Power Board v1.3.1 Final

I notice there is a 250ms gap between initial connection and TTFB. Is it because the Start Offset is 0.326s ? If so, is there something I can do to reduce this start offset time? I am on a 300MB VPS with very light CPU load (0.05~0.15) on a 100Mbps line. Where should I start looking? This has been bugging me for a long time, I just feel my site isn’t loading as fast as it should be. I even went through the code line by line.