His look of fear on his face makes me doubt his fondness of me..?

I think he's not scared of you. He's just nervous around you like trying to figure out whether he is your "type".