What about independently re-inventing the same thing, why should one have to pay royalties? You shouldn't, is the right answer.
Ideas freely come to people and should be given away freely to all mankind. There is no such thing as stealing an idea. Duplication is good. Amelioration (improving upon) is better. The old greek idea of progress was something like imitation, emulation, amelioration.
Patents are like putting fences around little pieces of land and forcing everyone who has to pass over it to reach the next place to pay up. Pretty soon, it is no longer economical to move anywhere, unless you are one of the big landlords, who make cross-agreements. The little people were unfree. That is feudalism, and it was a time of slavery. Only people living near coastlines or in big cities had some measure of freedom.
Patent only hinder people to get to the next idea, if you pay them any heed. Patents are modern feudalism in the space of ideas. Apple, IBM, Sony, Microsoft and others are the feudal lords in idea-space.
I think patents are unethical and immoral. Software, genetic and pharmaceutical patents are the worst of all.