Streets are an important component of urban landscapes and reflect the image, quality of life, and vitality of public spaces. With the help of the Google Cityscapes urban dataset and the DeepLab-v3 deep learning model, we segmented panoramic images to obtain visual statistics, and analyzed the impact of built environment attributes on a restaurant’s popularity. The results show that restaurant reviews are affected by the density of traffic signs, flow of pedestrians, the bicycle slow-moving index, and variations in the terrain, among which the density of traffic signs has a significant negative correlation with the number of reviews. The most critical factor that affects ratings on restaurants’ food, indoor environment and service is pedestrian flow, followed by road walkability and bicycle slow-moving index, and then natural elements (sky openness, greening rate, and terrain), traffic-related factors (road network density and motor vehicle interference index), and artificial environment (such as the building rate), while people’s willingness to stay has a significant negative effect on ratings. The qualities of the built environment that affect per capita consumption include density of traffic signs, pedestrian flow, and degree of non-motorized design, where the density of traffic signs has the most significant effect.