Yeah I think there is more to the Ben and Annie story. I hope we get to hear it before the series ends. Actually I hope we get to it before this season ends but I'm not gonna hold my breath.
One thing to keep in mind is this. When he first met long haired Richard in the jungle he told him that he hated the Dharma place and wanted Richard to take him away. He didn't say: "Well there is this girl Annie that I like, but everything else sucks." He just said he hated it all. So either he didn't care that much for Annie or at that point she was already gone.

__________________

Doctor playing golf. Woo, boy howdy, now I've heard everything. What's next, cop eating a donut?