I think it’s not usually worth worrying about.
If an app really needs an uninstaller for some reason, there’s a good chance that the developer either provides one, or explains how to uninstall it on their website.
For the vast majority of apps, the stuff they leave behind are small text files like preferences. Because there is no behemoth “registry” like Windows, there’s no downside to keeping these around. The amount of disk-space they use is minimal, and they don’t interfere with other apps, so I tend to just delete the app and leave that be it.
Hazel
If you want something to do generalized cleanup of those files, Hazel has a feature built-in called “App Sweep” (see the “Trash” tab in Hazel’s window).
If you move an app to the trash when Hazel is running (and Hazel should always be running), it will notice, and offer to move related files to the trash as well. If you move the app out of the trash, Hazel will offer to restore the files that it had moved to the trash.
I use that feature pretty much because I’m already using Hazel, and I can either choose yes or no when it asks me if I want to remove the related files. I probably leave them about 50% of the time, unless I’m absolutely sure that I’m never going to use the app ever again.