Time off on the calendar does not add hours to someone's utilization. It removes the availability from their timeline. So if you have a task that is scheduled for 2 days starting on Friday and Monday is a holiday, and the person assigned (and / or the project) has a calendar with that time not available, the scheduled end date will be Tuesday. People can still record time against Monday, but they will not show availability.
When you look at their allocation in the resource grid you'll also see that the total available will be 32 hours (assuming an 8 hour day) for that week.