I think you have somewhat similar noses and mouths. So I can imagine why your brother would say that. However, eyes and hairline are pretty different, so no chance I would confuse the two of you.
you look exactly like him!
Hey, no but you may favor him; next time lose the smoke in his picture and hold your head up in yours. You may look like cousins but not really much alike.