How far are you and your friend into the game's storyline - does the game indicate about 93% of progression for any of you?

Starting from around 93%, all the quests (including side quests) need to be completed in single-player only by design. Once you complete the main campaign to 100%, co-op will become unlocked again.

If, however, your quest progression is below 93% and you still cannot join your friend(s):

CLEAR THE CACHE on all consoles that are trying to join each other.

Additionally, make sure no other account is logged into your Xbox besides your own account.

Also, if you have your Internet router set to IPV6, try changing it to IPV4.

Lastly, if you have an external hard drive added to your console, uninstall the game, disconnect the drive and re-install the game to the console's default drive, please.

See also:

Co-op / Multiplayer Instructions

Network ports used by XboxOne

NAT settings