I think that physics is definitely more difficult. Physics is the application of mathematics and requires a basic understanding of mathematical terms and procedures. Math, however, does not require any prior understanding of physics. Likewise, making a comic book by yourself would be more difficult than making a cartoon by yourself. Making a whole comic book requires a basic understanding of drawing a simple cartoon but drawing a cartoon does not require understanding of making a whole comic book.
Simple steps are easier and once you build off of these foundations, subjects get more difficult.
Hope I helped!
physics, by far
math=physics. physics=math. need more specifics. long division is easier than physics. AP calculus is harder than basic physics.