Recent studies demonstrate that visual working memory capacity is greater for real-world objects compared to simple features like colors and scrambled objects. This led to the proposal that conceptual meaning plays a critical role in structuring visual working memory (Chung, Brady, & Störmer, 2024). However, one challenge in comparing memory performance across stimulus sets is that they vary not only in conceptual meaning but also in perceptual similarity. Thus, some of the working memory benefits for real-world objects may arise from these perceptual differences – for example whether the visual system interprets inputs as objects or not. Here, we provide a strong test of this by using novel objects generated by generative adversarial networks designed to resemble real objects (Cooper et al., 2023), and compare memory performance across novel and familiar real-world objects. Across experiments, participants remembered sets of four objects drawn from one of four stimulus types: familiar objects, novel objects, scrambled familiar objects or scrambled novel objects. After a short delay, they completed a two-alternative-forced-choice task, selecting between a target and a foil object. Results revealed enhanced working memory performance only for familiar objects, with no differences among the other conditions. Importantly, convolutional neural networks analyses confirmed comparable perceptual similarities between familiar and novel objects relative to scrambled stimuli. Thus, although novel objects closely resembled familiar objects, they did not enhance memory performance. Further correlation analyses revealed that subjective familiarity ratings are correlated with memory performance for familiar objects, while low-level features like colorfulness are correlated with memory performance for novel objects, suggesting that visual memory relies on different aspects to best remember each stimulus type. Overall, these results demonstrate that “object-ness” alone is insufficient to enhance visual working memory. Instead, familiarity and conceptual knowledge are critical in improving working memory performance.