Communication is an essential part of human life. For the person with hearing and speaking disability, it is inconvenient to communicate with other people. In this paper, an end-to-end system to convert English voice to Indian Sign Language (ISL) gloss (written form of sign language) is proposed which will help deaf to communicate with others and vice versa. This system accepts English voice as an input and, converts it into the text using the speech recognition. From the recognized English text, ISL gloss is generated using the lexical database called WordNet. The focus of our work is to build a robust sign language machine translation system to convert the English text to ISL gloss using the linguistic database WordNet.