Implicit discourse relation recognition (IDRR) aims to infer logical semantic relations between two adjacent text spans (also named arguments) without explicit indictive connectives, which is crucial to discourse analysis. Previous methods primarily focus on capturing the semantic features of discourse or the complex interaction patterns between the two arguments, including the utilization of pre-trained language models (PLMs). However, these approaches often overlook the fact that the semantic understanding of an argument cannot be interpreted independently from the overall paragraph-level discourse structure. Additionally, information about paragraph structure can help organize the context to establish semantic coherence in discourse from the perspective of cognitive linguistics, thereby creating a clear cognitive framework for the reader. Therefore, we propose a multi-hierarchy graph convolutional network framework based on paragraph-level discourse units for IDRR. Unlike conventional methods that utilize arguments as language units, we explore discourse units (DUs) within the broader context of a paragraph. Specifically, we employ the Stanford Parser to capture the syntactic dependency clues of each sentence within the DU and integrate them into the sentence representation learning. To further extract the structural information of paragraphs within the discourse, we leverage a multi-hierarchy graph convolutional network to derive the DU representation enriched with structural cues. Subsequently, we adopt multi-head attention to obtain important and structurally enhanced semantic features. The experimental results on the Penn Discourse TreeBank (PDTB) demonstrate that our model achieves performance comparable to that of benchmark models.