I want to break up with him, but I fear the outcome?

This relationship seems toxic, so you definitely should break up with him if you are not happy together. He'll get over it. Hoepfully people will grow up and not spread stupid lies around. Your ex's sound like weird people.