Physics is the fundamental study of Nature, in other words, in physics we want to find out how stuff works. Mathematics, on the other hand, is harder to pin down since it exists only in the human mind. Mathematics epitomizes the word "abstract." But still this can't be the complete story because many mathematical structures can be used to describe how physical objects interact and the nature of the relationships among them.
In physics we use mathematics as a tool to understand Nature. In mathematics, the pure notions of numbers and other structures do not need physics to exist or explain or even justify them. But the surprising thing is that often some newly discovered abstract formulation in mathematics turns out, years later, to describe physical phenomena which we hadn't known about earlier. When, as a student of physics, you see this for the first time, it is truly overwhelming. The only conclusion I can bring myself to is that mathematics is not just a tool of physics; it must be much, much more. Since mathematics is a product of our imagination, then somehow the structure of the universe itself seems to be imprinted on the human mind. And if that is the case, the relationship between mathematics and physics does indeed boil down to the chicken-and-the-egg question.