I think you did the right thing since he was there first. HOWEVER if it were me, I'd have given him my famous one finger salute and my two favorite words after his comment. You were being courteous, he was not.