Okay, turns out I misunderstood the way Avg. Time Spent on Site works. If there's a dimension attached, calculates it based on that dimension. I got confused because without a dimension (just using a segment for rows), instead of getting the overall Avg. Time Spent on Site for a whole visit for ever...