Should we actually divorce?

At this point, I would suggest seeing a marriage counselor to discuss your feelings of being secondary to the rest of his family even after five years of marriage. This sounds like a serious problem, but the two of you talking to a third party might be able to resolve it if he listens and comes to understand in the sessions how much this hurts you. It sounds like you are both still at a stage where you haven't withdrawn your feelings completely, and aren't totally ready to throw in the towel. It would help you both to learn to communicate in a constructive way about these issues. It's actually encouraging that you argue about it, because that means that at least you're communicating, but not in a helpful way yet. Just tamping down your hurt feelings will not solve the problem, and he's got to admit there's a problem before you can solve it.