1. How does the association of lead/person happens with mkt_tok? is this via the AssociateLead JS API?
No, the Munchkin associateLead method is for manual association using a special user-generated hash key (not the mkt_tok).
In contrast, mkt_tok association happens automatically when a Visit Web Page activity is logged.
2. Where is this mkt_tok stored in the Marketo DB against a lead record? I assume its a hashed key and a hidden field in DB for every lead/person (not supposed to be visible for security reasons). Also, do we really care to know about it?
You don't really need to know anything about it except that when it's missing, neither native pre-fill nor session auto-association will work.
It'll be missing, for example, when a link is completely untracked, either deliberately or because it was tokenized incorrectly in an email. (The mkt_tok query param can also be manually added to links that are not click-tracked, though this is a very uncommon need.)
There's also not a one-to-one relationship between mkt_tok and lead. Each mkt_tok is specific to an email send + lead combo. So "it" isn't stored but rather "they."
3. Where does the mkt_tok query param get appended to LP URL?
Where? Well, with a standard tracked link, after tracking the initial click (by rewriting the link to bounce off the tracking/branding domain) Marketo redirects the browser to the original URL w/the mkt_tok appended.