Sometimes agents and publishers list “literary” fiction as something they handle:
Fiction areas of interest: Action/Adventure, Experimental, Family Saga, Glitz, Historical, Humor, Literary, Mainstream, Mystery/Suspense, Religious, Thriller, Women’s.
Cruise the web and you will find various definitions of the term “literary fiction.” Some agents and publishers don’t even try to define it, saying simply that they know it when they see it.
Two of the five definitions of the adjective literary given in the OED are pertinent to the quest to define the term “literary novel.”
literary: Of or pertaining to, or of the nature of, literature. a. Pertaining to letters or polite learning. b. Pertaining to books and written compositions; also, in a narrower sense, pertaining to, or having the characteristics of that kind of written composition which has value on account of its qualities of form.
literary: Acquainted with or versed in literature.
According to the Wikipedia article headed “Literary fiction,”
Literary fiction is a term that has come into common usage since around 1960, principally to distinguish serious fiction (that is, work with claims to literary merit) from the many types of genre fiction and popular fiction. In broad terms, literary fiction focuses more on style, psychological depth, and character; the plot may or may not be important.
Several definitions I’ve seen indicate that literary novels are all character and reflection and no plot.
Literary agent Nathan Bransford insists that literary novels do have plots, but they are more difficult for the reader to discern:
In commercial fiction the plot tends to happen above the surface and in literary fiction the plot tends to happen beneath the surface.
The Wikipedia article goes on to use a term that’s new to me: paraliterature:
Paraliterature is an academic term for genre literature, such as science fiction, fantasy, mystery, pulp fiction and comic books, which is not generally considered literary fiction by mainstream literary standards.
The term smacks of condescension, as the prefix “para-” is used to form
miscellaneous terms in the sense ‘analogous or parallel to, but separate from or going beyond, what is denoted by the root word’.
“Paraliterature” is “like” literature, but not quite the real thing. Academic nonsense. Many genre writers produce literary fiction, mystery writer P.D. James, for example.
The 1960s as a beginning date for the term “literary fiction” is significant.
Rudolf Flesch’s expose of the abysmal failure of reading instruction in U. S. schools, Why Johnny Can’t Read and What You Can Do About It, was published in 1955. It shone light on the fact that Americans educated with the new “progressive” theories of reading instruction were not fluent readers.
Along with reduced reading skills came a decline of general knowledge. Before these declines, one definition of a novel was “a long fictional narrative written in literary prose.” Readers with reduced vocabularies and little general knowledge require stories that make fewer demands on comprehension. What used to be simply “novels,” are now categorized as “literary novels.”
Writers of “literary” fiction are well-read. They rejoice in extensive vocabularies acquired by reading the literary canon. They write for readers like themselves. When their characters and plots are sufficiently compelling, they astound the publishing world by attracting large numbers of readers who are willing to make the effort to read their books to the end.
Some examples of “literary” fiction that have appealed to large audiences and are on their way to becoming classics are
The Name of the Rose by Umberto Eco
The Time Traveler’s Wife by Audrey Niffenegger
One Hundred Years of Solitude by Gabriel Garcia Marquez
The Secret Life of Bees, by Sue Monk Kidd
The Lovely Bones, by Alice Sebold
The Handmaid’s Tale, by Margaret Atwood
Like any novel, a literary novel has an unfolding plot and characters that the reader can care about. What makes it “literary” is the background from which the writer writes. The vocabulary may be more extensive than common, and when the story ends, readers feel they’ve learned not just the fate of the characters, but something about the human condition.