前端智能化漫談 (1) - pix2code

自從有了GUI圖形界面，就誕生了跟圖形界面打交道的開發工程師，其中最大的一撥就演化成現在的前端工程師。不管是工作在前端、移動端還是桌面用戶端，跟界面布局和切圖等工作打交道是工作中的重要一部分。能夠直接從設計稿生成代碼，不僅是前端工程師的夢想，也是很多設計師同學的期望。

2017年，一篇名為《pix2code: Generating Code from a Graphical User Interface Screenshot》的論文橫空出世，立刻引發了廣泛關注。

pix2code是做什麼的

如下圖所示，pix2code通過将螢幕截圖與對應的DSL描述通過深度神經網絡進行訓練，然後給出一張新圖去進行推理得出一個新的DSL描述，最後再通過代碼生成器變成目标平台上的代碼。

前端智能化漫談 (1) - pix2code前端智能化漫談 (1) - pix2code

下面我們分别看下在Android平台，iOS平台和Web上的例子。

Android平台樣例

對應的DSL如下：

stack {
row {
btn, switch
}
row {
radio
}
row {
label, slider, label
}
row {
switch
}
row {
switch, switch, btn
}
row {
btn, btn
}
row {
check
}
}
footer {
btn-dashboard, btn-dashboard, btn-home
}

Web樣例

我們再看一個web的樣例：

header {
btn-inactive, btn-inactive, btn-inactive, btn-active, btn-inactive
}
row {
single {
small-title, text, btn-red
}
}

iOS平台樣例

stack {
row {
img, label
}
row {
label, slider, label
}
row {
label, switch
}
row {
label, btn-add
}
row {
img, label
}
row {
label, slider, label
}
row {
img, img, img
}
row {
label, btn-add
}
}
footer {
btn-search, btn-search, btn-download, btn-more
}

header {
btn-inactive, btn-inactive, btn-inactive, btn-active, btn-inactive
}
row {
single {
small-title, text, btn-red
}
}

pix2code的原理

pix2code的原理是将抓圖通過卷積網絡提取特征，同時将DSL通過LSTM循環神經網絡進行訓練，二者再統一放到一個循環神經網絡中進行訓練。

推理的時候，隻有抓圖進入卷積網絡，DSL序列為空，輸出結果為一DSL序列。

廢話不多說，直接上代碼。

一些參數

首先是一些配參數，比如輸入形狀，訓練輪數等：

CONTEXT_LENGTH = 48
IMAGE_SIZE = 256
BATCH_SIZE = 64
EPOCHS = 10
STEPS_PER_EPOCH = 72000

模型的儲存和讀取

訓練的網絡權值不能浪費了，AModel提供S/L大法：

from keras.models import model_from_json

class AModel:
    def __init__(self, input_shape, output_size, output_path):
        self.model = None
        self.input_shape = input_shape
        self.output_size = output_size
        self.output_path = output_path
        self.name = ""

    def save(self):
        model_json = self.model.to_json()
        with open("{}/{}.json".format(self.output_path, self.name), "w") as json_file:
            json_file.write(model_json)
        self.model.save_weights("{}/{}.h5".format(self.output_path, self.name))

    def load(self, name=""):
        output_name = self.name if name == "" else name
        with open("{}/{}.json".format(self.output_path, output_name), "r") as json_file:
            loaded_model_json = json_file.read()
        self.model = model_from_json(loaded_model_json)
        self.model.load_weights("{}/{}.h5".format(self.output_path, output_name))

卷積網絡

卷積網絡部分，6個卷積層分三段，最後是兩個1024節點的全連接配接網絡

image_model = Sequential()
        image_model.add(Conv2D(32, (3, 3), padding='valid', activation='relu', input_shape=input_shape))
        image_model.add(Conv2D(32, (3, 3), padding='valid', activation='relu'))
        image_model.add(MaxPooling2D(pool_size=(2, 2)))
        image_model.add(Dropout(0.25))

        image_model.add(Conv2D(64, (3, 3), padding='valid', activation='relu'))
        image_model.add(Conv2D(64, (3, 3), padding='valid', activation='relu'))
        image_model.add(MaxPooling2D(pool_size=(2, 2)))
        image_model.add(Dropout(0.25))

        image_model.add(Conv2D(128, (3, 3), padding='valid', activation='relu'))
        image_model.add(Conv2D(128, (3, 3), padding='valid', activation='relu'))
        image_model.add(MaxPooling2D(pool_size=(2, 2)))
        image_model.add(Dropout(0.25))

        image_model.add(Flatten())
        image_model.add(Dense(1024, activation='relu'))
        image_model.add(Dropout(0.3))
        image_model.add(Dense(1024, activation='relu'))
        image_model.add(Dropout(0.3))

        image_model.add(RepeatVector(CONTEXT_LENGTH))

        visual_input = Input(shape=input_shape)
        encoded_image = image_model(visual_input)

文本處理網絡

文本處理部分使用兩個LSTM

decoder = LSTM(512, return_sequences=True)(decoder)
        decoder = LSTM(512, return_sequences=False)(decoder)
        decoder = Dense(output_size, activation='softmax')(decoder)

        self.model = Model(inputs=[visual_input, textual_input], outputs=decoder)

        optimizer = RMSprop(lr=0.0001, clipvalue=1.0)
        self.model.compile(loss='categorical_crossentropy', optimizer=optimizer)

圖像和文本串聯在一起

圖像和文本都處理好之後，我們将其并聯在一起：

decoder = concatenate([encoded_image, encoded_text])

并聯好之後，我們再用另外兩級LSTM網絡來進行訓練。

decoder = LSTM(512, return_sequences=True)(decoder)
        decoder = LSTM(512, return_sequences=False)(decoder)
        decoder = Dense(output_size, activation='softmax')(decoder)

整個網絡建構好之後，我們就可以進行訓練了。

self.model = Model(inputs=[visual_input, textual_input], outputs=decoder)

        optimizer = RMSprop(lr=0.0001, clipvalue=1.0)
        self.model.compile(loss='categorical_crossentropy', optimizer=optimizer)

fit的過程中會将之前參數檔案中的EPOCHS和BATCH_SIZE讀進來，如下：

def fit(self, images, partial_captions, next_words):
        self.model.fit([images, partial_captions], next_words, shuffle=False, epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=1)
        self.save()

進行推理時的代碼如下：

def predict(self, image, partial_caption):
        return self.model.predict([image, partial_caption], verbose=0)[0]

    def predict_batch(self, images, partial_captions):
        return self.model.predict([images, partial_captions], verbose=1)

支援的DSL

最後我們看一下pix2code具體都支援哪些DSL元件。

Android DSL

Android平台支援16種DSL

其中對應到控件的有8個：

stack
row
label
btn
slider
check
radio
switch

輔助性的DSL4個：
opening-tag
closing-tag
body
footer

還有4種在FooterBar上的按鈕：
btn-home
btn-dashborad
btn-notifications
btn-search

與Android代碼的對應表如下：

{
  "opening-tag": "{",
  "closing-tag": "}",
  "body": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<LinearLayout\n    xmlns:android=\"http://schemas.android.com/apk/res/android\"\n    xmlns:app=\"http://schemas.android.com/apk/res-auto\"\n    xmlns:tools=\"http://schemas.android.com/tools\"\n    android:id=\"@+id/container\"\n    android:layout_width=\"match_parent\"\n    android:layout_height=\"match_parent\"\n    android:orientation=\"vertical\"\n    tools:context=\"com.tonybeltramelli.android_gui.MainActivity\">\n    {}\n</LinearLayout>\n",
  "stack": "<FrameLayout android:id=\"@+id/content\" android:layout_width=\"match_parent\" android:layout_height=\"match_parent\" android:layout_weight=\"1\" android:padding=\"10dp\">\n    <LinearLayout android:layout_width=\"match_parent\" android:layout_height=\"match_parent\" android:orientation=\"vertical\">\n        {}\n    </LinearLayout>\n</FrameLayout>",
  "row": "<LinearLayout android:layout_width=\"match_parent\" android:layout_height=\"wrap_content\" android:orientation=\"horizontal\" android:paddingTop=\"10dp\" android:paddingBottom=\"10dp\" android:weightSum=\"1\">\n{}\n</LinearLayout>",
  "label": "<TextView android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:text=\"[TEXT]\" android:textAppearance=\"@style/TextAppearance.AppCompat.Body2\"/>\n",
  "btn": "<Button android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:text=\"[TEXT]\"/>",
  "slider": "<SeekBar android:id=\"@+id/[ID]\" style=\"@style/Widget.AppCompat.SeekBar.Discrete\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:layout_weight=\"0.9\" android:max=\"10\" android:progress=\"5\"/>",
  "check": "<CheckBox android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:paddingRight=\"10dp\" android:text=\"[TEXT]\"/>",
  "radio": "<RadioButton android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:paddingRight=\"10dp\" android:text=\"[TEXT]\"/>",
  "switch": "<Switch android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:paddingRight=\"10dp\" android:text=\"[TEXT]\"/>",
  "footer": "<LinearLayout android:layout_width=\"match_parent\" android:layout_height=\"wrap_content\" android:orientation=\"horizontal\" android:weightSum=\"1\">\n    {}\n</LinearLayout>",
  "btn-home": "<Button android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:background=\"#0ffffff\" android:layout_weight=\"1\" android:drawableBottom=\"@drawable/ic_home_black_24dp\" android:text=\"\"/>",
  "btn-dashboard": "<Button android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:background=\"#0ffffff\" android:layout_weight=\"1\" android:drawableBottom=\"@drawable/ic_dashboard_black_24dp\" android:text=\"\"/>",
  "btn-notifications": "<Button android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:background=\"#0ffffff\" android:layout_weight=\"1\" android:drawableBottom=\"@drawable/ic_notifications_black_24dp\" android:text=\"\"/>",
  "btn-search": "<Button android:id=\"@+id/[ID]\" android:layout_width=\"wrap_content\" android:layout_height=\"wrap_content\" android:background=\"#0ffffff\" android:layout_weight=\"1\" android:drawableBottom=\"?android:attr/actionModeWebSearchDrawable\" android:text=\"\"/>"
}

iOS DSL

iOS有15個DSL項

常用控件6種：

img
輔助結構3個：
特殊結構6個：
btn-add
btn-contact
btn-download
btn-more

{
  "opening-tag": "{",
  "closing-tag": "}",
  "body": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<document type=\"com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB\" version=\"3.0\" toolsVersion=\"11201\" systemVersion=\"15G1217\" targetRuntime=\"iOS.CocoaTouch\" propertyAccessControl=\"none\" useAutolayout=\"YES\" useTraitCollections=\"YES\" colorMatched=\"YES\">\n    <dependencies>\n        <deployment identifier=\"iOS\"/>\n        <plugIn identifier=\"com.apple.InterfaceBuilder.IBCocoaTouchPlugin\" version=\"11161\"/>\n        <capability name=\"documents saved in the Xcode 8 format\" minToolsVersion=\"8.0\"/>\n    </dependencies>\n    <scenes>\n        <!--View Controller-->\n        <scene sceneID=\"qAw-JF-viq\">\n            <objects>\n                <viewController id=\"[ID]\" sceneMemberID=\"viewController\">\n                    <layoutGuides>\n                        <viewControllerLayoutGuide type=\"top\" id=\"[ID]\"/>\n                        <viewControllerLayoutGuide type=\"bottom\" id=\"[ID]\"/>\n                    </layoutGuides>\n                    <view key=\"view\" contentMode=\"center\" id=\"[ID]\">\n                        <rect key=\"frame\" x=\"0.0\" y=\"0.0\" width=\"375\" height=\"667\"/>\n                        <autoresizingMask key=\"autoresizingMask\" widthSizable=\"YES\" heightSizable=\"YES\"/>\n                        <subviews>\n                          {}\n                        </subviews>\n                        <color key=\"backgroundColor\" white=\"1\" alpha=\"1\" colorSpace=\"calibratedWhite\"/>\n                    </view>\n                </viewController>\n                <placeholder placeholderIdentifier=\"IBFirstResponder\" id=\"[ID]\" userLabel=\"First Responder\" sceneMemberID=\"firstResponder\"/>\n            </objects>\n            <point key=\"canvasLocation\" x=\"20\" y=\"95.802098950524751\"/>\n        </scene>\n    </scenes>\n</document>\n",
  "stack": "<stackView opaque=\"NO\" contentMode=\"center\" fixedFrame=\"YES\" axis=\"vertical\" alignment=\"center\" spacing=\"10\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" minX=\"16\" minY=\"20\" width=\"343\" height=\"440\"/>\n    <autoresizingMask key=\"autoresizingMask\" flexibleMaxX=\"YES\" flexibleMaxY=\"YES\"/>\n    <subviews>\n        {}\n    </subviews>\n    <color key=\"backgroundColor\" red=\"0.80000001190000003\" green=\"0.80000001190000003\" blue=\"0.80000001190000003\" alpha=\"1\" colorSpace=\"calibratedRGB\"/>\n</stackView>",
  "row": "<view contentMode=\"center\" ambiguous=\"YES\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" width=\"343\" height=\"65\"/>\n    <subviews>\n        <stackView opaque=\"NO\" contentMode=\"center\" fixedFrame=\"YES\" spacing=\"30\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n            <frame key=\"frameInset\" minX=\"8\" minY=\"6\" width=\"337\" height=\"52\"/>\n            <autoresizingMask key=\"autoresizingMask\" flexibleMaxX=\"YES\" flexibleMaxY=\"YES\"/>\n            <subviews>\n                {}\n            </subviews>\n        </stackView>\n    </subviews>\n    <color key=\"backgroundColor\" red=\"0.9\" green=\"0.9\" blue=\"0.9\" alpha=\"1\" colorSpace=\"calibratedRGB\"/>\n</view>",
  "img": "<imageView userInteractionEnabled=\"NO\" contentMode=\"scaleToFill\" horizontalHuggingPriority=\"251\" verticalHuggingPriority=\"251\" ambiguous=\"YES\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" width=\"36\" height=\"36\"/>\n    <color key=\"backgroundColor\" red=\"0.40000000600000002\" green=\"0.40000000600000002\" blue=\"1\" alpha=\"1\" colorSpace=\"calibratedRGB\"/>\n</imageView>",
  "label": "<label opaque=\"NO\" userInteractionEnabled=\"NO\" contentMode=\"left\" horizontalHuggingPriority=\"251\" verticalHuggingPriority=\"251\" ambiguous=\"YES\" text=\"[TEXT]\" textAlignment=\"natural\" lineBreakMode=\"tailTruncation\" baselineAdjustment=\"alignBaselines\" adjustsFontSizeToFit=\"NO\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" width=\"255\" height=\"52\"/>\n    <fontDescription key=\"fontDescription\" type=\"system\" pointSize=\"17\"/>\n    <nil key=\"textColor\"/>\n    <nil key=\"highlightedColor\"/>\n</label>",
  "switch": "<switch opaque=\"NO\" contentMode=\"scaleToFill\" horizontalHuggingPriority=\"750\" verticalHuggingPriority=\"750\" ambiguous=\"YES\" contentHorizontalAlignment=\"center\" contentVerticalAlignment=\"center\" on=\"YES\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" width=\"51\" height=\"31\"/>\n</switch>",
  "slider": "<slider opaque=\"NO\" contentMode=\"scaleToFill\" ambiguous=\"YES\" contentHorizontalAlignment=\"center\" contentVerticalAlignment=\"center\" value=\"0.5\" minValue=\"0.0\" maxValue=\"1\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" width=\"142\" height=\"31\"/>\n</slider>",
  "btn-add": "<button opaque=\"NO\" contentMode=\"scaleToFill\" ambiguous=\"YES\" contentHorizontalAlignment=\"center\" contentVerticalAlignment=\"center\" buttonType=\"contactAdd\" lineBreakMode=\"middleTruncation\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" width=\"22\" height=\"22\"/>\n</button>",
  "footer": "<tabBar contentMode=\"scaleToFill\" fixedFrame=\"YES\" translatesAutoresizingMaskIntoConstraints=\"NO\" id=\"[ID]\">\n    <frame key=\"frameInset\" height=\"49\"/>\n    <autoresizingMask key=\"autoresizingMask\" widthSizable=\"YES\" flexibleMinY=\"YES\"/>\n    <color key=\"backgroundColor\" white=\"0.0\" alpha=\"0.0\" colorSpace=\"calibratedWhite\"/>\n    <items>\n        {}\n    </items>\n</tabBar>",
  "btn-search": "<tabBarItem systemItem=\"search\" id=\"[ID]\"/>",
  "btn-contact": "<tabBarItem systemItem=\"contacts\" id=\"[ID]\"/>",
  "btn-download": "<tabBarItem systemItem=\"downloads\" id=\"[ID]\"/>",
  "btn-more": "<tabBarItem systemItem=\"more\" id=\"[ID]\"/>"
}

Web的DSL

Web的DSL有16種：

header
btn-active
btn-inactive
single
double
quadruple
btn-green
btn-orange
btn-red
big-title
small-title
text

{
  "opening-tag": "{",
  "closing-tag": "}",
  "body": "<html>\n  <header>\n    <meta charset=\"utf-8\">\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n    <link rel=\"stylesheet\" href=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css\" integrity=\"sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u\" crossorigin=\"anonymous\">\n<link rel=\"stylesheet\" href=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css\" integrity=\"sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp\" crossorigin=\"anonymous\">\n<style>\n.header{margin:20px 0}nav ul.nav-pills li{background-color:#333;border-radius:4px;margin-right:10px}.col-lg-3{width:24%;margin-right:1.333333%}.col-lg-6{width:49%;margin-right:2%}.col-lg-12,.col-lg-3,.col-lg-6{margin-bottom:20px;border-radius:6px;background-color:#f5f5f5;padding:20px}.row .col-lg-3:last-child,.row .col-lg-6:last-child{margin-right:0}footer{padding:20px 0;text-align:center;border-top:1px solid #bbb}\n</style>\n    <title>Scaffold</title>\n  </header>\n  <body>\n    <main class=\"container\">\n      {}\n      <footer class=\"footer\">\n        <p>&copy; Tony Beltramelli 2017</p>\n      </footer>\n    </main>\n    <script src=\"js/jquery.min.js\"></script>\n    <script src=\"js/bootstrap.min.js\"></script>\n  </body>\n</html>\n",
  "header": "<div class=\"header clearfix\">\n  <nav>\n    <ul class=\"nav nav-pills pull-left\">\n      {}\n    </ul>\n  </nav>\n</div>\n",
  "btn-active": "<li class=\"active\"><a href=\"#\">[]</a></li>\n",
  "btn-inactive": "<li><a href=\"#\">[]</a></li>\n",
  "row": "<div class=\"row\">{}</div>\n",
  "single": "<div class=\"col-lg-12\">\n{}\n</div>\n",
  "double": "<div class=\"col-lg-6\">\n{}\n</div>\n",
  "quadruple": "<div class=\"col-lg-3\">\n{}\n</div>\n",
  "btn-green": "<a class=\"btn btn-success\" href=\"#\" role=\"button\">[]</a>\n",
  "btn-orange": "<a class=\"btn btn-warning\" href=\"#\" role=\"button\">[]</a>\n",
  "btn-red": "<a class=\"btn btn-danger\" href=\"#\" role=\"button\">[]</a>",
  "big-title": "<h2>[]</h2>",
  "small-title": "<h4>[]</h4>",
  "text": "<p>[]</p>\n"
}

前端智能化漫談 (1) - pix2code前端智能化漫談 (1) - pix2code

前端智能化漫談 (1) - pix2code

pix2code是做什麼的

Android平台樣例

Web樣例

iOS平台樣例

pix2code的原理

一些參數

模型的儲存和讀取

卷積網絡

文本處理網絡

圖像和文本串聯在一起

支援的DSL

Android DSL

iOS DSL

Web的DSL

繼續閱讀

方法數超了65535 無法安裝的解決方案

android調用系統相機拍攝擷取照片的兩種方法

Android調用系統拍照擷取圖檔問題

測試工程師面試解析～

iOS開發之送出App前需要準備的材料

iOS APP送出上架最新流程

Android Studio中Make Project、Clean Project、Rebuild Project 的作用

新買的便攜式藍牙标簽列印機內建測試起因分析提供的SDK開始使用藍牙測試工具進行測試列印列印亂碼分析JAVA中Unicode轉ANCI工具連結

2018年不想被web前端開發淘汰，你需要掌握哪些技術？

K-近鄰算法以及圖像分類應用

Android面試題——5.常見面試官提問Android面試題①

Android常用工具類簡介目錄

android常用工具類轉載自：http://www.trinea.cn/android/android-common-utils/Android常用工具類

EGORefreshTableHeaderView 解讀代碼解讀 ELTableViewController 的使用寫在最後

今日頭條iOS用戶端啟動速度優化技術調研實測資料

Android電視機（機頂盒）初次開發的一些經驗分享