The baby monitoring system is a kind of smart IoT system helpful for busy working parents. Besides providing basic safety care, it actually also delivers the moments of baby growing and development by all time recording. Hence, in baby monitoring systems nowadays, capturing and recommending the precious moments of infant growth to their parents has become a real-world demand. How to capture the interesting, worthwhile and exquisite moments with commemorative and catering to the parents' preferences is a challenging problem. In this paper, we propose a novel image rating framework, namely Infant growth Precious Moment capturing (IPreMom), which is based on the concept of continual learning for comprehensively addressing the problem of automatically capturing the precious moments during infant growth. Through a series of experiments, it was shown that our proposed framework delivers excellent performance compared to the baselines, with up to 40% improvement in terms of MAE. Moreover, the proposed framework has been implanted into a commercial baby monitor on the market. To the best of our knowledge, this is the first work that solves the problem of automatically capturing the infant precious moments using continual learning and image rating techniques, which is important and has not been well studied in the academic and industrial communities.