I'm sure app.execMenuItem("Close") works fine when running the form in Reader or Acrobat outside of the browser but within a browser, I think you would run into problems such as the one you're describing.
If your form will always be viewed within a browser, how about simply navigating to some "exit" URL? As a matter of fact, you could create a little HTML file that has a "window.close" call in its initialization script and point the browser to that URL when the user clicks on the button like this:
app.launchURL("close.html");
This might have the effect you're looking for within a browser window.
Stefan
Adobe Systems