Perhaps the answer lies within its meaning. The internet defines literature as "written works, especially those considered of superior or lasting merit" with Collins English Dictionary adding "written material… especially works of imagination characterized by excellence of style and expression and by themes of general and enduring interest".This appears to be where the problem is, though on the face of things there's no reason that should exclude fantasy. To many, literature, as a form of art, must imitate life and as fantasy for the most part is not set in our world it is left out of this mindset. There are, however, numerous problems with this.
Some of the oldest stories humans have been telling are based in fantasy. Legends and mythology are the staples of what led to the genre as we can see in The Lord of the Rings - it reads as a Christianised version of the Norse tale of Ragnarok. Considering we are still seeing re-workings of Greek legends such as The Song of Achilles by Madeline Miller (winner of the 2012 Orange Prize) these stories are clearly still striking a chord, despite their reliance on gods, demigods and centaurs. It must be said the fantasy elements were not what the book was praised for, instead focusing on the themes of friendship and love the book was imbued with. Yet, the fantasy elements were still there and, it would seem, a literary audience enjoyed them. Still on the theme of legends and mythology, as some of the oldest stories our species told, their structure has stayed with us and is still present in contemporary fiction. Most stories, including particularly literary works, still follow the traditional three-act structure. To say that fantasy does not have “lasting artistic merit” would be to say that neither does the origin of fiction - something of an oxymoron.