1. Can we track the action performed on the links in the email based on the email id's(as its not part of recipient table)
Under Storage tab of External file target mapping you can see message schema is linked with Recipient delivery logs and tracking logs is linked with Recipient tracking logs. So, even if you do not have that email ID in recipient table but still there will be one record for this email ID in nms:broadLogRcp and trackinglogRcp. So yes, you can track action performed on the links in the email.
2. Drawbacks of targeting the ppl directly from file which is not part of recipient table
It's just you will not be able to use columns defined in recipient table 🙂