For the sake of making it interesting, I added a sling:orderBefore property to the custom-button node to illustrate how you would place your button in the list wherever you would like
How to overlay the right way
You have overlaid the whole /libs/wcm/core/content/editor node which not a good idea. Here is a picture of your node tree, with the area you actually need to change (in green) vs what you've overlaid but didn't actually need to (in red):
As you can see, this is not very efficient, but more importantly, it is actually a bit dangerous You can read more about overlays here and more about how Sling manages them here, from the official Adobe documentation
By comparison, look at my node tree:
And in this whole tree, only the last node contains properties, the rest are empty and simply serve to create the path from apps to custom-button. This is much more efficient (both in time and in the size of the /apps node) and more future-proof.
To illustrate my point, lets imagine the following scenario.
I don't know what version of AEM you are on but image that you do an upgrade in the future which adds functionalities and/or fixes to the editor by modifying some nodes and files under /libs/wcm/core/content/editor. You would never benefit from those changes, because despite them being installed under /libs, when you load the editor, Sling will first check in /apps, find a resource and never look in /libs.
As a best best practice, you should aim to overlay the absolute minimum. Don't overlay nodes unless you need to change their properties and don't overlay properties unless you really need to make a change. Not only that, but each time you upgrade, you should be patching all your customizations! If you have overlaid 500 nodes vs 50, that will multiply the complexity of your upgrades by 10.
I hope this has helped you? If you have any more questions or if anything in my answer is unclear, please don't hesitate to ask