Embedding user's portrait in smart library's mobile visual search service can solve the information overload and the semantic gap problem, and meet the personal needs of the readers. The paper made a review of the related research of mobile library visual search and user portrait, explored the realization process of user portrait in data collection、data preprocessing、modeling and effect evaluation of smart library mobile visual search and recommendation service, constructed personalized mobile visual search and recommendation service model, probed different function of the model in data collection, resources integration, data processing, and smart service, and analyzed some key problems such as privacy protection, dynamic changes of the portrait, and optimization of service.