Does he still have a crush on her?

My boyfriend and I are in a long distance relationship and early on he told me that he used to have a huge crush on his friend's older sister. He said 'used to' so I didn't bring up the matter again. Not too long after that he went to stay over at his friend's house and we were talking on aim. I hadn't talked to him at all that day so I was looking forward to it. But after a few minutes he didn't respond after awhile. I was a bit annoyed but I just went and did something else to keep myself busy. About 3 hours later he im'd me and told me that he was sorry and that his friend's older sister had come over and they were talking, watching movies, etc. I let it go because I figure they're fairly good friends and he doesn't get to see her often. However, he's done this several times. Right in the middle of conversation when we've barely talked, he'll just leave. It just makes me feel like he literally just tosses me aside anytime she comes over- I'd honestly be happy with a 'I'll brb in a couple hours' but he doesn't even do that. He just leaves for a few hours and leaves me in the dark wondering what happened or what he's doing. I am trying to take into consideration that he doesn't see her that often, but I still feel like he drops everything and puts me on hold anytime she's around.It really seems like I'm put in second place. Am I exaggerating or should I be a bit concerned?