As I see it, formalism is best defined in terms of the philosophy of Immanuel Kant, where it is closely linked with the term "autonomy." Kant employs the term "autonomy" in his ethical philosophy by contrast with the term "heteronomy." Autonomy in ethics means, above all, that an ethical action must be undertaken for its own sake, not for some ulterior motive, for the sake of a reward, or with a view to the consequences of the action. The ethical act is motivated only by itself, through consideration of pure duty. This is why Kant refers to his own ethical philosophy as "formalist" in character: it is not primarily a matter of the good or bad "content" of an action, but of whether it was motivated by ethical consideration alone.
While "formalism" and "autonomy" are not mentioned by Kant in his theory of art, they lie implicitly at the heart of that theory. Kant's aesthetics are formalist insofar as beauty is distinguished both from what is personally agreeable and from the useful (as in Kant's own low estimation of architecture as an art form). The beautiful is what is beautiful in its own right, and given sufficiently developed taste, everyone ought to agree as to what is beautiful. In this way, the beauty of an artwork or anything else is autonomous from the various impure human motives that often taint our pure aesthetic judgment. And insofar as such judgment comes from the transcendental faculty that belongs universally to all humans, it pertains to us rather than to the beautiful work. Thus, there is an autonomy of the work and the human spectator from each other. [...] OOO greatly prefers Kant's formalist aesthetics to others of a Hegelian, Marxist, or Frankfurt School inspiration, in which art belongs to a dynamic interplay of the entire realm of spirit, so that the artwork reflects numerous historical factors rather than being cleanly separated from them.
Harman goes on to outline the approach he takes in his book, which "argues that both the good and bad side of Kantian formalism are adopted by the modernist formalism represented brilliantly by Clement Greenberg and Michael Fried; their rejection of the term "formalism" is based on a different definition of the term from the one adopted in this book." That's not so interesting to me, but by all means read the rest of his post for yourself.
I'm not at all sure what I think of Kantian formalism (as Harman outlines it). It exists in a different intellectual world from the one I work in. My primary interest in form is as a descriptive category, certainly for all forms of art, but I am particularly interested in literary art. How that relates to aesthetics, I don't at the moment know.
I do note, however, that I am interested in art having a measure of autonomy from "numerous historical factors", though I have no problem seeing it as "reflecting" them. It has no choice but to be intertwined with the historical world, but art nonetheless as it's own resources which aren't so implicated. Those resources are to be located in the resources and pressures of human biology and the role it plays in cultural evolution. I lay some groundwork for such a position in the final section of my 2006 paper on literary morphology [1]. It's there in my current working paper on the graph in Jockers' Macroanalysis in the section "Literary culture is a force in history", pp. 17-19 [2].
