Since "Average Page Load Time" isn't a standard metric, I am assuming you are using the plugin "getPageLoadTime"....
I don't use this myself, but I would first check that the plugin is returning expected results during testing... Have you confirmed that the data collected does look correct when you are viewing the product pages?
If the data looks like its tracking correct, I would then start with a more basic report... it's possible that the way the data is being segmented could be causing some unintended oddities? Start with the most basic table, confirm that values look realistic, then one by one, layer in segmentation and see if you can determine where things might be going wrong... double check how the segment(s) is/are created, maybe it (they) need(s) to be tweaked?