UIBee: An improved deep instance segmentation and classification of UI elements in wireframes

Abstract

User Interface (UI) is a basic concept in which individuals interact with any computer program or technological device to create a graphical design. In the initial stages of app development, UI prototype is a must. An automatic analysis system for the basic execution of UI designs will considerably speed up the development of designs according to old-fashioned methods. In this approach, it is aimed at saving cost and time by automating the process. For the aforesaid objective, we present a new approach rather than the traditional methods. For this reason, a high amount of elements in wireframes are detected and segmented. Furthermore, with the state-of-the-art methods, one of the machine learning classifiers is expected to give lower performance than deep learning for comparison purposes. In this study, the detection and segmentation of elements, which is the first stage which will eliminate time loss, redundant time, cost, and labor in the communication between designers and front-end developers. To test the classification task of the Mask R-CNN, was designed using transfer learning supported neural networks to compare with other algorithms. As a result, the precision reached 93.15% and the mAP (@IOU > 0.5) reached 96.50%. Then, we improved the algorithm by replacing the convolution blocks in the graphs, adding them, and changing the input units, and the accuracy increased to 98.49%.