If this is a caching issue, it is probably related to the statfileslevel  in your dispatcher configuration. The dispatcher invalidates certain parts of the content tree in the cache based on this configuration. The general recommendation is to set the statfileslevel as high as possible but as low as necessary for your website. This may somehow be conflicting with the Sitemap XML as it is usually placed at the root of the website.
There are different ways to handle this, depending on the overall structure of your website:
Move the sitemap.xml to a higher level in the content hierarchy so that it lies within the invalidated area covered through the statfileslevel setting. You can manage the URL to the sitemap through rewrite rules on the dispatcher and still expose it externally on the root level.
User ACS AEM Commons Dispatcher Flush Rules  to define a mapping that invalidates the sitemap whenever a page of your website is invalidated.
Usually not the best solution: decrease the statfileslevel until it also invalidates your sitemap xml.
If this is not a caching issue it is hard to give advice without any knowledge about your implementation of the sitemap xml generation.