How do I not get so irritated with the way my boyfriend parents?

Lately, I ahve been very irritated with the way my boyfriends parents. We both have children (mine are 3 and 6 and his is 3) and I have never dated a guy with kids before so I feel like I need to get use to my children not being the only kids in the pitcure. We are really serious, we;ve discussed marriage and living together. I am really apprehensive with the live-in situation because I get really irritated with his son. I feel horrible about it because I know he is just a little kid but I can't stop feeling this way. My boyfriend seems to baby him so he whines a lot. Compared to my three year old he seems very babyfied. My three year old even tells me that he doesn't like to play with him because he doesn't share and he cries alot. I tell my son that that's mean of him to say and to try and get along but I feel terrible because deep down he irritates me too. I want to get close to him but I don't know how to overcome the fact that I'm annoyed. Also, I always want to step in a discipline him because my boyfriend doesn't give him consequences for not listening to him. He doesn't have a set bed time so he can't spend the night for the simple fact that my kids do. We talked about living together but until we're on the same page I don't feel like it's such a good idea. Am I in the wrong?