We propose the first implementation of an infinite-order generative dependency model. The model is based on a new recursive neural network architecture, the Inside-Outside Recursive Neural Network. This architecture allows information to flow not only bottom-up, as in traditional recursive neural networks, but also top-down. This is achieved by computing content as well as context representations for any constituent, and letting these rep-resentations interact. Experimental re-sults on the English section of the Uni-versal Dependency Treebank show that the infinite-order model achieves a per-plexity seven times lower than the tradi-tional third-order model using counting, and tends to choose more accurate parses in k-best lists. In addition, reranking with this model achieves state-of-the-art unla-belled attachment scores and unlabelled exact match scores. 1