I don't think it would be insane, but
Madison is too much of a surname for me to like it on anyone, boy or girl. The sound somehow doesn't appeal to me either. However, due to its meaning, I would much prefer to see
Madison used for a boy if it must be used at all.