Theimplicit discourse relation classification is of great importance to discourse analysis. It aims to identify the logical relation between sentence pair. Compared with the linear network model, the graph neural network has a more complex structure to capture cross-sentence interactions. Therefore, this article proposes a semantic graph neural network for implicit discourse relation classification. Specifically, we design a semantic graph to describe the syntactic structure of sentences and semantic interactions between sentence pair. Then, convolutional neural network (CNN) with different convolutional kernels to extract the multi-granularity semantic features. The experimental results on Penn Discourse TreeBank 2.0 (PDTB 2.0) prove that our work performed well.