We have a custom_publish_farm.any which support many virtualhosts domains and working fine till I have added a new farm(customfrag_publish_farm.any) for handling requests for a new domain.
However after adding I'm observing lot of requests getting 500 internal error and dispatcher.log and modsec_audit.log size is growing fast.
Even after reverting back my changes which is deleting the new publish farm still lot of 500 errors in the dispatcher.log file. We are going to analyze more but want to check if adding any new farm will impact existing publish farm configurations or any dispatcher settings and break things.
As @asutosh_jena already pointed out, please verify your configuration syntax and double check on the logs of Apache httpd and dispatcher (error.log, dispatcher.log, other custom logs if there are any).
In general, adding a new farm should not do any harm (assuming that the configuration of the farm is correct). However, there may be situations where simple changes may break something. I've seen httpd no longer starting up after adding another vhost due to various reasons, e. g. exceeded open files limits or duplicate, invalid configuration items coming in with a new domain. On the other hand, these kind of errors usually result in httpd being not able to (re-) start and wouldn't lead to HTTP 500 responses.
You need to first check where the HTTP 500 is caused. Is it the dispatcher/httpd that has an internal error and is not able to fulfill the request? Or does the dispatcher get an according response from the publish instance and just forwards it? (Double check on the access.log and request.log of your publish instances.) Once you've found the source you should see some traces to follow further.
As you're mentioning mod_security: that log file is probably growing because it catches all the HTTP 500 responses and logs them. Still - under certain circumstances - mod_security may also cause HTTP 500 responses. You would find these in the mod_security-log file(s).