Start at the pink arrow on the left and ask yourself, are either of you related to this person? After that, elements such as family drama and closeness are factors. If it's not a relative, do you work or go to school with the person? Regularly spending time with someone is an indication that they should be invited, whereas, whether or not you were invited to their wedding can also be a factor.
All in all, keep your invitation list as long or short as you want - as long as you and your groom are happy with your selection, that's all that matters!