The modulation of Autonomic Nervous System (ANS) dynamics is a fundamental aspect of emotional response. However, the dynamics of the interplay between ANS activity and subjective perception of emotional states are still an open question. In this preliminary study, we explored the causal relationship between continuously annotated emotional ratings and physiological time series. We used a subset of the publicly-available Continuously Annotated Signals of Emotions (CASE) dataset, focusing our analysis on time-varying self-assessed arousal ratings (AR), electrocardiographic signals, and electrodermal activity (EDA), simultaneously acquired from 30 healthy participants during two different video-based emotional stimuli, i.e. scary and relaxing. We applied the Convergent Cross Mapping (CCM) approach to investigate the causal links between EDA, AR, and heart rate variability (HRV) time series. The results were compared against surrogate series, generated using the Amplitude Adjusted Fourier Transform, to assess their significance. Regardless of the stimulation type, a statistically significant <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathbf{p}-\mathbf{value} < \mathbf{1}\mathbf{e}-\mathbf{14})$</tex> causal inference was found between the HRV and both the EDA and AR signals, and a statistically significant <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathbf{p}-\mathbf{value} < \mathbf{1}\mathbf{e}-\mathbf{18})$</tex> bidirectional coupling was observed between EDA and AR. Our findings support previous literature about physiologically-driven emotional processes, testing, for the first time, this hypothesis on the three-variable system of real-time arousal, HRV, and EDA recordings.