I found the solution for this problem:
Issue #1 : Bundle with new version was not getting installed:
The issue was related to an older version of "org.apache.sling.installer.factory.configuration". I installed the newer version of this bundle from <host>:<port>/system/console using the "Install/Update" option at the top right and restarted the instance.
Old version was 1.2.1 and now i have installed 1.3.4.
Issue #2: Old bundle was present and if we were deleting it, it was getting reflected after server restart
We checked the bundle in crx-quickstart/launchpad/felix and it was pointing to non existing jar file. So we have deleted this bundle after taking the backup of AEM.
After server restart, bundle was gone from the server.