I always thought the worldbuilding Narnia was a little subpar, but that's a classic, so... But yes, if your goal is to write something fun, fast, and even possibly thought provoking for reasons that have nothing to do with that facet of worldbuilding, great, that's your choice, but don't be surprised if your book is a flash in the pan.

I always say I read for the characters (and the politics!) but that's not entirely true. If the worldbuilding is too weak, it will throw me out of the story , and then I won't get to enjoy my characters. And the way the worldbuilding doesn't work in Harry Potter is making it very difficult for me to write AU fic for it.