In conversations, interlocutors with hearing impairment (HI), initiate their turns with longer average delay and more variability than their normal-hearing (NH) conversation partners. This paper investigates whether third-party listeners were able to perceive this difference in turn-taking timing (denoted Floor-Transfer Offsets, FTOs) between NH and HI interlocutors. To this avail, the FTOs of segments of conversations were manipulated using four different schemes, such that the two interlocutors timed their turns as if 1) both were NH interlocutors conversing in quiet (NHQ) or 2) both were HI interlocutors conversing in noise (HIN). Two additional conditions were implemented to test if non-varying and faster FTOs affect the perception, manipulating conversations to have constant FTOs corresponding to 3) the median FTO of the NHQ condition (conNHQ) and 4) an even faster FTO of 50 ms (conLow). Forty-four participants rated conversations from the HIN condition as less natural, with poorer flow, and being more difficult to follow compared to conversations from the NHQ condition. Removing the FTO variability resulted in better ratings of flow (conNHQ vs NHQ), while further decreasing the average FTOs (conLow vs conNHQ) had no effect on the ratings. A detailed analysis revealed poorer ratings of flow for individual conversations with higher FTO median, but similar variability, while FTO variability alone did not affect ratings of flow. Together, the results indicate that listeners, and perhaps also conversation participants, are sensitive to small variations in turn-taking timing and that hearing loss may therefore affect the quality of conversations.