The Lost Combinator

Let me begin by thanking the Association for Computational Linguistics and its Executive Committee for conferring on me the great honor of their Lifetime Achievement Award for 2018, which of course I share with all the wonderful students and colleagues that have made many essential contributions to this work over many years.At the heart of the work that I have been pursuing over my research lifetime so far, whether in parsing and sentence processing, spoken language understanding, semantics, or even in musical understanding by machine, there lies a theory of natural language grammar that brings parsing, compositional semantics, statistical modeling, and logical inference into the closest possible relation. This theory of grammar is combinatory, in the sense that its operations are type-dependent and restricted to strictly string-adjacent phonologically or graphologically-realized inputs, and categorial, in the sense that those operands pair a syntactic type with a type-transparent semantic representation or logical form.I'd like to use this opportunity to briefly address three questions that revolve around the theory of grammar, both combinatory and otherwise. The first question concerns the way that Combinatory Categorial Grammar (CCG) was developed with a number of colleagues, over a number of stages and in slightly different forms. The second is an essentially evolutionary question of why natural language grammar should take a combinatory form. The third question is that of what the future holds for CCG and other structural theories of grammar in computational linguistics and NLP in the age of deep learning.I have called this talk "The Lost Combinator" in homage to the Victorian era poem "The Lost Chord," in the hope of suggesting that the theoretical development of CCG has always been empirical, rather than axiomatic, in search of the simplest explanation of the facts of language, rather than for confirmation of linguistic received opinion, however intuitively salient.In the late 1960s (when I was a psychology undergraduate at the University of Sussex under Stuart Sutherland, and then started as a graduate student in artificial intelligence at Edinburgh under Christopher Longuet-Higgins), a broad community of theoretical linguists, psychologists, and computational linguists saw themselves as all working on the same problem, under the definition provided by the "transformational" theory of grammar proposed by Chomsky (1957, 1965), using theories of psycholinguistic processing, language acquisition, and language evolution proposed by Lashley (1951), Miller, Galanter, and Pribram (1960), Miller (1967), and Lenneberg (1967), theories of natural language semantics proposed by Carnap (1956), Montague (1970), and Lewis (1970), and computational models of parsing such as those proposed by Thorne, Bratley, and Dewar (1968) and Woods (1970). (I myself was so convinced that this program would succeed that I believed it was time to apply the same methods to other cognitive faculties, taking as my research project for Ph.D. their application to the interpretation of music by machine, following the lead of Max Clowes [1971] in machine vision.)Almost immediately, this consensus fell apart. First, Chomsky himself was among the first (1965) to recognize that transformational rules, though descriptively revealing, were so expressive as to have little explanatory force, and required many apparently arbitrary constraints (Ross 1967). Second, psychologists realized that psycholinguistic measures of processing difficulty of sentences bore almost no relation to their transformational derivational complexity (Marslen-Wilson 1973; Fodor, Bever, and Garrett 1974). Finally, computational linguists attempting to implement transformational grammars as parsers realized that they were spending all their time implementing even more constraints on rules, in order to limit search arising from overgeneration (Friedman 1971; Gross 1978). (Meanwhile, I realized that the problem had not in fact been solved, and returned to natural language processing, thanks to a postdoc at Sussex with Philip Johnson-Laird.)This disillusion wasn't just a case of internal academic squabbling. There were also a couple of influential reports commissioned by the U.S. and UK governments that ended funding for machine translation (MT) and artificial intelligence (AI) (Pierce et al. 1966; Lighthill 1973). As a result of the second of these reports, which determined that AI was never going to work, PhDs in artificial intelligence like my classmate Geoff Hinton and myself spent ten years or so after graduation in psychology departments (in my case, at the Universities of Sussex and Warwick), until yet another report said AI was working after all and that Britain and the U.S. were falling behind Japan in this vital area. As a result, I could get hired again in computer science, first briefly back at Edinburgh, and then at the University of Pennsylvania (I learned a lesson from this odyssey that I have tried to remember whenever I have been appointed to a committee to report on anything, which is that while reports very rarely do any good, they can very easily do a great deal of harm.)Meanwhile, as a result of these conflicts, the scientific study of language fragmented. The linguists swiftly abjured any responsibility for their grammars ("Competence") bearing any relation to processing ("Performance"). Because the psychologists could hardly abandon Performance, they in turn became agnostic about grammar, retreating to context-free surface grammar (which they tended to refer to as "parsing strategies"), or a touchingly optimistic belief in its emergence from neural models. Meanwhile, the computational linguists (whose machines were growing exponentially in size and speed from the 16K byte core of the machine that supported the whole group when I started my graduate studies, on to levels that would soon permit parsing the entire contents of the then embrionic Web) similarly found that very little of what the linguists and psychologists cared about was usable at scale, and that none of it significantly improved overall performance over very much simpler context-free or even finite-state methods that the linguists had shown to be incomplete. The reason of course was Zipf's law, which means that the events with respect to which the low-level methods are incomplete are off in the long tail.It also became apparent to a few computationalists working on speech, MT, and information retrieval that the real problem was not grammar but ambiguity and its resolution by world-knowledge, and that the solution lay in probabilistic models (Bar-Hillel 1960/1964; Spärck Jones 1964/1986; Wilks 1975; Jelinek and Lafferty 1991) (although it was not immediately apparent how to combine statistical models with grammar-based systems without making obviously false independence assumptions).Nevertheless, as any red-blooded psychologist had always insisted, the divorce between competence and performance that everyone else had accepted did not make any sense. The grammar and the processor had to have evolved in lock-step, as a package deal, for what could be the evolutionary selective advantage of a grammar that you cannot process, or a parser without a grammar?It seemed equally obvious that surface syntax and the underlying semantic or conceptual representation must also be closely related, since the only reasonable basis for child language acquisition that has ever been on offer is that the child attaches language-specific grammar to a universal conceptual relation or "language of mind" (Miller 1967; Bowerman 1973; Wexler and Culicover 1980). It seemed to follow that radically new theories of grammar were needed.Theoretical linguists agree that the central problem for the theory of grammar is discontinuity or non-adjacent dependency between predicates and their arguments:Chomsky described discontinuity in terms of movement, which was known to be formally very unconstrained. By contrast, the ATN parser used in the LUNAR project (Woods, Kaplan, and Nash-Webber 1972) reduced all discontinuity to local operations on registers (Thorne, Bratley, and Dewar 1968; Bobrow and Fraser 1969; Woods 1970).In particular, unbounded wh-dependencies like the above were handled by: (a) putting a pointer into a * or HOLD register as soon as the "which" was encountered without regard to where it would end up; and (b) retrieving the pointer from HOLD when the verb needing an object "had" was encountered without regard to where it had started out. (It also included an ingenious mechanism for coordination called SYSCONJ, which one finds even now being reinvented on an almost yearly basis—cf. Woods [2010].) A * register was also used for wh-constructions within a systemic grammar framework by Winograd (1972, pages 52–53) in his inspiring conversational program SHRDLU.However, it was unclear how to generalize the HOLD register to handle the multiple long-range dependencies, including crossing dependencies, that are found in many other languages. In particular, if the HOLD register were assumed to be a stack, then the ATN becomes a two-stack machine (since we are already implicitly using one stack as a PDA to parse the context-free core grammar).On the computational side at least, the reaction to this impass took two distinct forms. Both reactions took the form of trying to reduce the two major operators of the transformation theory, substitution of immediate constituents, or what is nowadays called "Merge," and "Move," or displacement of non-immediate constituents, to one. On the one hand, Lexical Functional Grammar (Bresnan and Kaplan 1982) and Head-driven Phrase Structure Grammar (Pollard and Sag 1994) followed Kay (1979) in making unification the basis of movement and merger. Because unification can pass information across unbounded structures, this can be thought of as reducing Merge to Move.On the other hand, Generalized Phrase Structure Grammar (Gazdar 1981), Tree Adjoining Grammar (TAG; Joshi and Levy 1982), and Combinatory Categorial Grammar (CCG, Ades and Steedman, 1982) sought to reduce Move to various forms of local merger. In particular, the latter authors suggested that the same stack could be used to capture both long-range dependency and recursion in CCG.1Natural language grammar exhibits discontinuity because semantically language is an applicative system. Applicative systems (such as programming languages) support the twin notions of: (a) Application of a function/concept to an argument/entity; and (b) Abstraction, or the definition of a new function/concept in terms of existing ones.Language is in that sense inherently computational. It seems to follow that linguistics is (or should be) inherently computational as well. (Of course, it does not follow that computationalists have nothing to learn from linguistics.)There are two ways of modeling abstraction in applicative systems: Taking abstraction itself as a primitive operation (λ-calculus, LISP):(2)a.fatherEsau⇒Isaacb.grandfather=λx.father(fatherx)c.grandfatherEsau⇒Abrahamor Defining abstraction in terms of a collection of operators on strictly adjacent terms aka Combinators, such as function composition (Combinatory Calculus, MIRANDA).(3)b′.grandfather=BfatherfatherThe latter does the work of the λ-calculus without using any variables.Despite the resemblance of the "traces" (or copies) and "operators" (or complementizer positions) of the transformational theory to the λ-operators and variables of applicative systems of the first kind, natural language actually seems to be a system of the second, combinatory kind. The evidence stems from the fact that natural language deals with all sorts of fragments that linguists do not normally think of as semantically typable constituents, without the use of any phonologically realized equivalent of variables, such as pronouns:(4)a.Give[Anna books]?and[Manny records]?b.(Mother to child): There's adoggie![Youlike]?#the doggie.c.Food that you must[washVP/NP[before eating](VP∖VP)/NP]?.d.ik denk dat ik1Henk2Cecilia3[zag1leren2zingen3]?These fragments are diagnostic of a Combinatory Calculus based on Bn, T, and the "duplicator" Sn, plus application (Steedman 1987; Szabolcsi 1989; Steedman and Baldridge 2011).2CCG lexicalizes all bounded dependencies, such as passive, raising, control, exceptional case-marking, and so forth, via lexical logical form. All syntactic rules are Combinatory—that is, binary operators over contiguous phonologically realized categories and their logical forms. These rules are restricted by a Combinatory Projection Principle, which in essence says they cannot override the decisions already taken in the language-specific lexicon, but must be consistent with and project unchanged the directionality specified there. All such language-specific information is specified in the lexicon: The combinatory rules like composition are free and universal. All arguments, such as subjects and objects, are lexically type-raised to be functions over the predicate, as if they were morphologically cased as in Latin, exchanging the roles of predicate and argument.All long-range dependencies are established by contiguous reduction of a wh-element, such as (N∖N)/(S/NP), with an adjacent non-standard constituent with category S/NP, formed by rules of function composition.The combinatory rules synchronize composition of the syntactic types shown here with corresponding composition of logical forms (suppressed in the derivations above), to yield the logical forms shown as λ-terms for the resulting nouns N.To capture the construction in Example (4c), whose syntactic derivation we pass over here, we also need rules based on the duplicator S:(7)a."wash X before eating X″b.VP/NP:λx.before(eatx)(washx)≡S(Bbeforeeat)washTo capture constructions like Example (4d) (whose syntactic derivation is similarly suppressed), we also need rules based on second-order composition B2:(8)a."Y saw X teach W to sing.″b.((S∖NP)∖NP)∖NP:λwλxλy.help(teach(singw)wx)xy≡B2sees(Bteachsing)CCG thus reduces the operator move of transformational theory to applications of purely adjacent operators—that is, to recursive combinatory merge.Interestingly, the latest "minimalist" form of the transformational theory has also proposed that Move should be relabeled as an "internal" form of standard or "external" Merge (Chomsky 2001/2004, page 110), though without providing any formal basis for the reduction other than identifying internal Merge as "a grammatical transformation." (If anything deserved the soubriquet "the lost combinator," it would be this notional unitary combination of application and abstraction in a single perfect operator, linking or merging all types, as in the epigraph to this article.)B2 rules allow us to "grow" categories of arbitrarily high valency, such as ((S∖NPy)∖NPx)∖NPw. As we saw earlier, in some Germanic languages like Dutch, Swiss German, and West-Flemish, serial verbs are linearized using such rules to require crossing discontinuous dependencies. Thus, B2 rules give CCG slightly greater than context-free power.Nevertheless, CCG is still not as expressive as movement. In particular, we can only capture permutations that are what is called "separable," where separability is related to the idea of obtaining the permutations by rebracketing and rotating sister nodes (Steedman 2018).For example, for the categories of the form A|B, B|C, C|D, and D, it is obvious by inspection that we cannot recognize the following permutations:(9)i.*B|CDA|BC|Dii.*C|DA|BDB|CThis generalizatiion appears likely to be true cross-linguistically for the components of this form for the NP "These five young boys":(10)i.*Five boys these youngii.*Young these boys fiveTwenty-one of the 22 separable permutations of "These five young boys" are attested (Cinque 2005; Nchare 2012). The two forbidden orders are among the unattested three.3The probability of this happening by chance is the probability of the is the of the number of ways of two of three unattested by the is, about one in a (If the order that unattested so were to be this chance would to about one in number of separable permutations much more in than the number of all example, for around of the permutations are There are obvious for the problem of in machine translation and neural semantic parsing, to which we I to and at first as a still under least, that was what Joshi and then as a of where much of the development of CCG was out. students in a of and Joshi 1987; and that the of Ades and Steedman was by that both CCG and were equivalent to Grammar (Gazdar a new of the by the and of languages by these fell within the of what Joshi called which proposed as a for what could as a theory of natural that they are and and some limit on crossing dependencies. the is much much than the including the multiple free languages and even the languages of so it seems to and CCG as with to the its CCG was assumed to be as a grammar for parsing, because of the derivational ambiguity by and the combinatory rules, these also under and as any grammar with the same as CCG the same of in the parser it is there in the In this is just another in the of derivational ambiguity that all natural language and can be handled by the same statistical models as other particular, the dependency models by and are and Steedman and CCG is also to parsing with which can be using and long and Steedman and is now used in those that for between semantic and syntactic processing, such as machine translation and and machine and parsing and Steedman 1987; and et al. et al. et al. and semantic parser and 2005; et al. et al. of my work with the same has returned to their application in musical and and Steedman and shown that CCG grammars of the same and parsing models of the same statistical are required there as It is only in the of their compositional semantics that music and language very than on this of work in like to by two more The first is an evolutionary should natural language be a combinatory in the first The second is a question about the future development of CCG and other grammar-based theories to be to NLP in the age of deep and recursive neural take these questions in order in the two language like a combinatory applicative system because T, and evolved in to support of before there was any language (Steedman like need and are for of need to to make can form the that you need to before is to that they allow and some other can with like B2 are to make with arbitrary of including and including other whose is yet to be to be to do the the and problem, using like to in order to that are of to of and so such as of and can apply to a and much work that the and other have to be there in the already for the to be to with was to that from the even if had the problem of can be as the problem of search for a of in a or of possible such search has the same recursive as parser example, there are both and for the The latter a more in evolutionary as a mechanism that is to both semantic interpretation and parsing, rather than the evolution of like the the for linguistic as as the operators for competence grammar, the two to as what was to above as an evolutionary hope to have convinced you that CCG grammars are both and as as semantic parsers that are to parse the like other CCG parsers are by the of parsing models based on only a of no how we using and the constructions they are dependencies and so as we have and off in the long As a parsers are to on performance overall by using models et al. says more about the of grammars and parsers than about the of deep in the of semantic parser for arbitrary such as and Steedman I would that CCG and other grammar-based parsers have already been by of deep neural and and the question of whether models for applications in are because we have to the universal semantic that allow the child to CCG for natural languages and that we to be using both in semantic parser and in Because the language is any of linguistic logical like the universal language of it be more with and to semantic parsers for by deep neural force, rather than by CCG semantic parser is it possible that the problem of parsing could be by neural by stack et al. et al. actually learn as has been seems likely that semantic parsers and neural machine translation to have difficulty with long-range because the evidence for their is so example, both and as a case, verb categories or a complementizer I is actually by which says that if you are an language with like and then you not in be to in languages like or languages like and German, you be to both subjects and objects, or at the time of a translation system no of learned these syntactic from a we get an sentence whose to translation means is the that the us the a is the that the us to the if we with we (which back into the a is the that the us holds the a we with using a the translation again an that is is the that they said had the is the they said they the contrast, CCG parsers do rather on and Steedman Steedman, and which is a construction that to be rather determined by the parsing methods are similarly when with long-range even when the sentence is in this think that the the and that it is et think that the the and it is in at least, these constructions could be learned by grammar-based semantic parser from using the methods of et al. and et al. make it likely that there be a need for in like where long-range dependencies like deep and are here to The future in parsing for such lies with systems using neural for and grammars for problem in NLP the fact that natural language understanding inference as as semantics, and we have no idea of the representation question is the almost it many it is almost equally to the information in a form that is not immediately with the form of the example, sentences like the following a different a rather than a an an a a and at a representation language that is we are using CCG parsers to the for between in order to consistent of between over of the same types, using over then an it and it under such as et al. of in the then that can be to a single relation and Steedman can be across from multiple languages and Steedman can then the semantics for relation with the and the entire using this now both and semantic an with the as and the as questions the in this we parse questions into the same semantics which is now the language of the the we use the and the and the following anything that in the then the the of anything is in the in the this to work, we need to be nodes in their in both and then be to like in of a semantic in the language of the need to learn between semantic and the language of the this project is the the function of a of the semantic in semantic like those of and and Steedman and while the form a similarly of the of Carnap and Fodor, Fodor, and Garrett semantic are essentially but with the advantage that they can be with logical operators such as and for the of semantic underlying natural language semantics in to another different to semantics that to use reduced of to using operations such as and and in of compositional It is an question whether can be with to of a and et al. It is likely that some of be here to the of the of the long like and work in do they work in In particular, can they learn all the syntactic in the long like and crossing in a way that support semantic they are not actually but are a finite-state or a then by on as for natural language processing, we are in of of the computational linguistic project of also providing computational of language and if we that like is a real and that learn their first language by of the sentences of their language the of the universal language of we still the of what that universal semantic language not get an to that question we can above using and and such as as for the language of to use machine for what it is such variables and their for use in a natural language work was supported in by a Award and a a University of Edinburgh and my and the and all my and students over many