Shall I be honest With my BF…????????

Q:Hello everyone. This question may sound terrible but it is doing my head in.. I have been with my boyfriend for 6 months and in the last month or so he keeps asking me is my penis small??? To be honest I dont care about his penis he is amazing!! I know he is kind of small probably around 4.5 maybe 4.75 inches but when he asks if it is small I say “dont be silly” and he gets annoyed!! so shall I be honest with him??? Not easy! has anyone else had same experience??