I agree with all of that. Ben did seem to not really care about the Swan, he called it "a joke". I think it was a surprise to all the Others when the hatch imploding caused such significant fallout on their communications and ability to travel on and off the island.