I love museums but my husband cannot spend nearly as much time roaming the halls as I can. We'll often begin together but then split up in our exploration. This way, the person waiting doesn't feel like they are rushing the slower person and the slower person doesn't feel like they are holding up the faster one. Sometimes, I just go on my own. He makes plans to see a site that I am not as interested in while I go to a museum. Then, we set a time and place to meet up again (This is key if you don't have an international phone and have trouble finding wifi to connect).
Splitting up is not as easy if you are traveling with younger children (at least my baby isn't quite ready to explore on his own). If you have multiple children, they may be in different stages of life, with very different interests. Parents may need to divide and conquer. One parent can stay with the younger children for nap times while the other parent explores a park or museum with older children.
When the kids are older, they may have stronger opinions about what they want to do on vacation. Giving them a say in the planning can be part of the fun. If your daughter loves animals, schedule horseback riding on the beach one afternoon. If your son is into art, try to find a street art tour on another day. Don't forget to do something that you love too! Giving everyone a say in the trip planning makes them more amenable to participating in all the activities.
Do you spend every moment of every day with your family? No. You don't have to do everything together when you are traveling either. If you do separate, make sure everyone is together for dinner. It's a great time to tell each other what you did and hearing how much family members enjoyed their day, always makes people happy. Everyone has different interests and the best vacations try to incorporate something for every member of the family.