通路www.arxivdaily.com擷取含摘要速遞，涵蓋CS|實體|數學|經濟|統計|金融|生物|電氣領域，更有搜尋、收藏、發帖等功能！點選閱讀原文即可通路
cs.CL 方向，今日共計39篇
Transformer(2篇)
【1】 Ad Text Classification with Transformer-Based Natural Language Processing Methods
标題：基于轉換器的自然語言處理方法在廣告文本分類中的應用
作者：Umut Özdil,Büşra Arslan,D. Emre Taşar,Gökçe Polat,Şükrü Ozan
機構：∗AdresGezgini A.¸S. Ar-Ge Merkezi,˙Izmir, Türkiye, †Dokuz Eylül Üniversitesi Y.B.S Y.L. Ögr., ˙Izmir, Türkiye, Özet—Bu çalı¸smada, çevrimiçi reklam platformlarında olu¸s-, turulan, metinlerinin, göre, otomatik
備注：6 pages, in Turkish language, 4 figures, 3 tables, 25. Pazarlama Konferans{\i} (25th Marketing Conference)
連結：https://arxiv.org/abs/2106.10899
摘要：本文提出了一種基于自然語言處理的線上廣告文本自動分類方法。我們的資料集由來自12個不同行業的約21000條貼标廣告文本組成。在本研究中，我們使用了一種基于Transformer的雙向編碼器表示（BERT）模型，這是一種最近在自然語言處理文獻中用于文本分類等領域的基于Transformer的語言模型。使用預先訓練好的BERT模型對土耳其語的分類效率進行了詳細的說明。
摘要：In this study, a natural language processing-based (NLP-based) method is proposed for the sector-wise automatic classification of ad texts created on online advertising platforms. Our data set consists of approximately 21,000 labeled advertising texts from 12 different sectors. In the study, the Bidirectional Encoder Representations from Transformers (BERT) model, which is a transformer-based language model that is recently used in fields such as text classification in the natural language processing literature, was used. The classification efficiencies obtained using a pre-trained BERT model for the Turkish language are shown in detail.
【2】 Transformers for Headline Selection for Russian News Clusters
标題：俄羅斯新聞叢集标題選擇的Transformer
作者：Pavel Voropaev,Olga Sopilnyak
機構：Moscow Institute, of Physics and Technology, Moscow, Russia
備注：Accepted to Dialogue 2021 conference
連結：https://arxiv.org/abs/2106.10487
摘要：在本文中，我們探讨了各種多語種和俄語預訓練Transformer為基礎的模型，為對話評估2021年共同的任務标題選擇。實驗結果表明，該方法優于單獨的多語言和單語模型。我們分析了幾種擷取句子嵌入的方法，并在此基礎上學習了一個排名模型。對于公共測試集和私有測試集，我們分别獲得了87.28%和86.60%的準确率。
摘要：In this paper, we explore various multilingual and Russian pre-trained transformer-based models for the Dialogue Evaluation 2021 shared task on headline selection. Our experiments show that the combined approach is superior to individual multilingual and monolingual models. We present an analysis of a number of ways to obtain sentence embeddings and learn a ranking model on top of them. We achieve the result of 87.28% and 86.60% accuracy for the public and private test sets respectively.
QA|VQA|問答|對話(1篇)
【1】 Learning to Rank Question Answer Pairs with Bilateral Contrastive Data Augmentation
标題：基于雙邊對比資料增強的問答對排序學習
作者：Yang Deng,Wenxuan Zhang,Wai Lam
機構：The Chinese University of Hong Kong
連結：https://arxiv.org/abs/2106.11096
摘要：在這項工作中，我們提出了一種新穎且易于應用的資料擴充政策，即雙邊生成（BiG），其目的是通過對比訓練來提高現有标記資料對問答對排序的性能。具體來說，我們通過兩個預先訓練的生成模型（一個用于問題生成，另一個用于答案生成）來合成僞正QA對，這兩個模型在原始資料集中有限的正QA對上進行微調。利用增廣的資料集，我們設計了一個對比訓練目标來學習問題-答案對的排序。在TREC-QA、WikiQA和ANTIQUE三個基準資料集上的實驗結果表明，該方法充分利用了已有的标記資料，顯著提高了排序模型的性能，并且可以友善地應用于不同的排序模型。
摘要：In this work, we propose a novel and easy-to-apply data augmentation strategy, namely Bilateral Generation (BiG), with a contrastive training objective for improving the performance of ranking question answer pairs with existing labeled data. In specific, we synthesize pseudo-positive QA pairs in contrast to the original negative QA pairs with two pre-trained generation models, one for question generation, the other for answer generation, which are fine-tuned on the limited positive QA pairs from the original dataset. With the augmented dataset, we design a contrastive training objective for learning to rank question answer pairs. Experimental results on three benchmark datasets, namely TREC-QA, WikiQA, and ANTIQUE, show that our method significantly improves the performance of ranking models by making full use of existing labeled data and can be easily applied to different ranking models.
機器翻譯(1篇)
【1】 Challenges in Translation of Emotions in Multilingual User-Generated Content: Twitter as a Case Study
标題：多語言使用者生成内容中情感翻譯的挑戰--以Twitter為例
作者：Hadeel Saadany,Constantin Orasan,Rocio Caro Quintana,Felix do Carmo,Leonardo Zilio
機構：Centre for Translation Studies, University of Surrey, Constantin Or˘asan, Roc´ıo Caro Quintana, RGCL, University of Wolverhampton, F´elix do Carmo
連結：https://arxiv.org/abs/2106.10719
摘要：盡管情感是一個普遍的概念，但是将不同的情感從一種語言轉移到另一種語言對于人類譯者來說并不總是那麼簡單，更不用說機器翻譯系統了。此外，認知狀态是由語言和文化語境共同塑造的對經驗的語言解釋來建立的。在許多言語語境中，情感的表達構成了資訊的關鍵組成部分。這對于使用者生成的内容（UGC）尤其如此，它可以是産品或服務的評論、tweet或社交媒體文章的形式。最近，像推特這樣的多語種網站通常會提供教資會的自動翻譯，以接觸不同語種的使用者。在這種情況下，翻譯使用者情緒的過程是完全自動的，沒有人為幹預，既不用于後期編輯，也不用于準确性檢查。在這項研究中，我們評估了自動翻譯工具是否能夠成功地在真實生活中傳遞使用者生成的多語言資料（如tweets）中的情感。我們發現Twitter資料中有一些特定的語言現象對不同語言中情感的翻譯提出了挑戰。我們在一系列語言特征中總結了這些挑戰，并展示了這些特征在不同語言對中的出現頻率。我們還評估了評價機器翻譯系統在保持原文情感方面的性能的常用方法的能力。
摘要：Although emotions are universal concepts, transferring the different shades of emotion from one language to another may not always be straightforward for human translators, let alone for machine translation systems. Moreover, the cognitive states are established by verbal explanations of experience which is shaped by both the verbal and cultural contexts. There are a number of verbal contexts where expression of emotions constitutes the pivotal component of the message. This is particularly true for User-Generated Content (UGC) which can be in the form of a review of a product or a service, a tweet, or a social media post. Recently, it has become common practice for multilingual websites such as Twitter to provide an automatic translation of UGC to reach out to their linguistically diverse users. In such scenarios, the process of translating the user's emotion is entirely automatic with no human intervention, neither for post-editing nor for accuracy checking. In this research, we assess whether automatic translation tools can be a successful real-life utility in transferring emotion in user-generated multilingual data such as tweets. We show that there are linguistic phenomena specific of Twitter data that pose a challenge in translation of emotions in different languages. We summarise these challenges in a list of linguistic features and show how frequent these features are in different language pairs. We also assess the capacity of commonly used methods for evaluating the performance of an MT system with respect to the preservation of emotion in the source text.
語義分析(2篇)
【1】 STEP-EZ: Syntax Tree guided semantic ExPlanation for Explainable Zero-shot modeling of clinical depression symptoms from text
标題：STEP-EZ：句法樹引導的語義解釋對臨床抑郁症狀文本的可解釋零射模組化
作者：Nawshad Farruque,Randy Goebel,Osmar Zaiane,Sudhakar Sivapalan
機構：Sivapalan, Department of Computing Science, University of Alberta, Alberta Machine Intelligence Institute (AMII), University of Alberta, Department of Psychiatry, University of Alberta
連結：https://arxiv.org/abs/2106.10928
摘要：我們緻力于探索Zero-Shot學習（ZSL）的各種方法，以及它們對一個因訓練資料匮乏而臭名昭著的具有挑戰性但又很重要的有監督學習任務的解釋能力，即從文本中檢測抑郁症狀（DSD）。我們首先在臨床醫生的幫助下，對ZSL模型的不同組成部分進行綜合，并對我們的基本事實樣本和抑郁症症狀線索的治療過程進行分析。接下來，我們将分析各種最先進的ZSL模型的準确性以及它們對我們任務的潛在增強。此外，我們還為使用ZSL進行基于文本的分層解釋機制勾畫了一個架構，我們稱之為文法樹指導的語義解釋（STEP）。最後，我們總結了實驗結果，從中我們可以得出結論，我們可以使用ZSL模型，并達到合理的準确性和解釋性，衡量了提出的解釋性指數（EI）。據我們所知，這項工作是第一次從準确性和解釋性兩個方面全面探讨ZSL模型在DSD任務中的有效性。
摘要：We focus on exploring various approaches of Zero-Shot Learning (ZSL) and their explainability for a challenging yet important supervised learning task notorious for training data scarcity, i.e. Depression Symptoms Detection (DSD) from text. We start with a comprehensive synthesis of different components of our ZSL modeling and analysis of our ground truth samples and Depression symptom clues curation process with the help of a practicing clinician. We next analyze the accuracy of various state-of-the-art ZSL models and their potential enhancements for our task. Further, we sketch a framework for the use of ZSL for hierarchical text-based explanation mechanism, which we call, Syntax Tree-Guided Semantic Explanation (STEP). Finally, we summarize experiments from which we conclude that we can use ZSL models and achieve reasonable accuracy and explainability, measured by a proposed Explainability Index (EI). This work is, to our knowledge, the first work to exhaustively explore the efficacy of ZSL models for DSD task, both in terms of accuracy and explainability.
【2】 A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic loss
标題：淺談語義缺失的生成性對話模式的訓練效果
作者：Prasanna Parthasarathi,Mohamed Abdelsalam,Joelle Pineau,Sarath Chandar
機構： School of Computer Science, McGill University , University of Montr´eal, ´Ecole Polytechnique de Montr´eal, Quebec Artificial Intelligence Institute (Mila), Canada CIFAR AI Chair
備注：Accepted at SIGDial 2021
連結：https://arxiv.org/abs/2106.10619
摘要：在對話任務中，為下一代話語訓練的神經模型學習模仿訓練集中的n-gram序列，訓練目标為負對數似然（NLL）或交叉熵。這種常用的訓練目标并不能促進對特定環境的替代反應。但是，最小化替代訓練目标對生成替代反應并對其進行評分的模型對語義相似度的影響還沒有得到很好的研究。我們假設一個語言生成模型可以通過在訓練過程中學習生成交替的文本并将語義損失最小化作為輔助目标來改善其多樣性。我們在兩個不同大小的資料集上探讨了這一觀點，并以目标導向對話中的下一代話語為研究對象。我們做了兩個觀察：（1）最小化一個語義目标在較小的資料集（幀）中改進了響應的多樣性，但隻與最小化較大資料集（MultiWoZ）中的NLL一樣好；（2）大型語言模型嵌入作為語義丢失目标比作為令牌嵌入的初始化更有用。
摘要：Neural models trained for next utterance generation in dialogue task learn to mimic the n-gram sequences in the training set with training objectives like negative log-likelihood (NLL) or cross-entropy. Such commonly used training objectives do not foster generating alternate responses to a context. But, the effects of minimizing an alternate training objective that fosters a model to generate alternate response and score it on semantic similarity has not been well studied. We hypothesize that a language generation model can improve on its diversity by learning to generate alternate text during training and minimizing a semantic loss as an auxiliary objective. We explore this idea on two different sized data sets on the task of next utterance generation in goal oriented dialogues. We make two observations (1) minimizing a semantic objective improved diversity in responses in the smaller data set (Frames) but only as-good-as minimizing the NLL in the larger data set (MultiWoZ) (2) large language model embeddings can be more useful as a semantic loss objective than as initialization for token embeddings.
Graph|知識圖譜|Knowledge(5篇)
【1】 Toward Knowledge Discovery Framework for Data Science Job Market in the United States
标題：美國資料科學就業市場的知識發現架構
作者：Mojtaba Heidarysafa,Kamran Kowsari,Masoud Bashiri,Donald E. Brown
機構： Department of Systems and Information Engineering, University of Virginia, Office of Health Informatics and Analytics, University of California, Los Angeles, School of Data Science, University of Virginia
連結：https://arxiv.org/abs/2106.11077
摘要：資料科學領域的發展需要更好的工具來了解這樣一個快速增長的領域。此外，來自不同背景的人開始對從事資料科學家的職業感興趣。是以，為個人群組織提供一個量化的指南，以了解就業市場所需的技能将是至關重要的。本文介紹了一個分析美國資料科學相關工作崗位就業市場的架構，同時提供了一個了解該市場的界面。提出的架構包括三個子子產品，允許連續資料收集、資訊提取和基于web的儀表闆可視化，以調查資料科學相關工作和技能的時空分布。這項工作的結果顯示了資料科學工作的主要分支的重要技能，并試圖為這些資料科學分支提供一個基于技能的定義。該應用程式的目前版本部署在web上，允許個人和機構通過行業視角調查資料科學職位所需的技能。
摘要：The growth of the data science field requires better tools to understand such a fast-paced growing domain. Moreover, individuals from different backgrounds became interested in following a career as data scientists. Therefore, providing a quantitative guide for individuals and organizations to understand the skills required in the job market would be crucial. This paper introduces a framework to analyze the job market for data science-related jobs within the US while providing an interface to access insights in this market. The proposed framework includes three sub-modules allowing continuous data collection, information extraction, and a web-based dashboard visualization to investigate the spatial and temporal distribution of data science-related jobs and skills. The result of this work shows important skills for the main branches of data science jobs and attempts to provide a skill-based definition of these data science branches. The current version of this application is deployed on the web and allows individuals and institutes to investigate skills required for data science positions through the industry lens.
【2】 Extractive approach for text summarisation using graphs
标題：一種基于圖的文本摘要抽取方法
作者：Kastriot Kadriu,Milenko Obradovic
機構：University of Ljubljana, Veˇcna pot , SI-, Ljubljana, Slovenia
備注：4 pages, 2 figures, 5 tables
連結：https://arxiv.org/abs/2106.10955
摘要：自然語言處理是一門重要的學科，其目的是通過文本的數字表示來了解文本，但由于我們的書寫和說話方式的多樣性，往往不夠準确。本文探讨了不同的圖形相關算法，可用于解決文本摘要問題的提取方法。我們考慮兩個量度：句子重疊和編輯距離來衡量句子相似度。
摘要：Natural language processing is an important discipline with the aim of understanding text by its digital representation, that due to the diverse way we write and speak, is often not accurate enough. Our paper explores different graph-related algorithms that can be used in solving the text summarization problem using an extractive approach. We consider two metrics: sentence overlap and edit distance for measuring sentence similarity.
【3】 ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction
标題：ROPE：基于圖的文檔資訊抽取的閱讀順序等變位置編碼
作者：Chen-Yu Lee,Chun-Liang Li,Chu Wang,Renshen Wang,Yasuhisa Fujii,Siyang Qin,Ashok Popat,Tomas Pfister
機構：†Google Cloud AI, §McGill University, ‡ Google Research
備注：Accepted to ACL-IJCNLP 2021 (Oral)
連結：https://arxiv.org/abs/2106.10786
摘要：自然的單詞閱讀順序對于從類似表單的文檔中提取資訊至關重要。盡管最近圖卷積網絡（GCNs）在文檔空間布局模式模組化方面取得了一些進展，但它們在捕獲給定單詞級節點表示的讀取順序方面能力有限。我們提出閱讀順序等變位置編碼（ROPE），這是一種新的位置編碼技術，旨在了解文檔中單詞的順序表示。ROPE為給定單詞級圖連通性的目标單詞相關的相鄰單詞生成唯一的讀取順序代碼。在公共FUNSD資料集和大規模支付資料集上，我們研究了兩個基本的文檔實體抽取任務，包括單詞标注和單詞分組。我們表明，ROPE持續改進現有的GCNs，其F1分數的內插補點高達8.4%。
摘要：Natural reading orders of words are crucial for information extraction from form-like documents. Despite recent advances in Graph Convolutional Networks (GCNs) on modeling spatial layout patterns of documents, they have limited ability to capture reading orders of given word-level node representations in a graph. We propose Reading Order Equivariant Positional Encoding (ROPE), a new positional encoding technique designed to apprehend the sequential presentation of words in documents. ROPE generates unique reading order codes for neighboring words relative to the target word given a word-level graph connectivity. We study two fundamental document entity extraction tasks including word labeling and word grouping on the public FUNSD dataset and a large-scale payment dataset. We show that ROPE consistently improves existing GCNs with a margin up to 8.4% F1-score.
【4】 JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs
标題：JointGT：用于從知識圖生成文本的圖文聯合表示學習
作者：Pei Ke,Haozhe Ji,Yu Ran,Xin Cui,Liwei Wang,Linfeng Song,Xiaoyan Zhu,Minlie Huang
機構：The CoAI group, Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology
備注：ACL 2021 (Findings)
連結：https://arxiv.org/abs/2106.10502
摘要：現有的知識圖到文本（KG-to-text）生成的預訓練模型隻是對文本到文本的預訓練模型進行微調，例如在KG-to-text資料集上的BART或T5，這在很大程度上忽略了編碼過程中的圖結構，并且缺乏詳細的預訓練任務來顯式地模組化圖-文本對齊。為了解決這些問題，我們提出了一個稱為JointGT的圖文聯合表示學習模型。在編碼過程中，我們設計了一個結構感覺的語義聚合子產品，嵌入到每個轉換層中，以保持圖的結構。此外，我們提出了三個新的預訓練任務來顯式地增強圖-文本對齊，包括各自的文本/圖重建，以及通過最優傳輸在嵌入空間進行圖-文本對齊。實驗表明，JointGT在不同的KG-to-text資料集上獲得了最新的性能。
摘要：Existing pre-trained models for knowledge-graph-to-text (KG-to-text) generation simply fine-tune text-to-text pre-trained models such as BART or T5 on KG-to-text datasets, which largely ignore the graph structure during encoding and lack elaborate pre-training tasks to explicitly model graph-text alignments. To tackle these problems, we propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation module which is plugged into each Transformer layer to preserve the graph structure. Furthermore, we propose three new pre-training tasks to explicitly enhance the graph-text alignment including respective text / graph reconstruction, and graph-text alignment in the embedding space via Optimal Transport. Experiments show that JointGT obtains new state-of-the-art performance on various KG-to-text datasets.
【5】 Enhancing Question Generation with Commonsense Knowledge
标題：利用常識知識促進問題生成
作者：Xin Jia,Hao Wang,Dawei Yin,Yunfang Wu
機構：†MOE Key Lab of Computational Linguistics, School of EECS, Peking University, ‡Baidu Inc., China
備注：Accepted by CCL2021
連結：https://arxiv.org/abs/2106.10454
摘要：問題生成（QG）是在給定的語境中生成自然的、文法的問題，這些問題可以由特定的答案來回答。以往的序列到序列模型存在一個問題，即提出高品質的問題需要以常識知識為背景，在大多數情況下不能直接從訓練資料中學習，導緻不滿意的問題被剝奪了知識。本文提出了一個多任務學習架構，将常識知識引入到問題生成過程中。我們首先從成熟的資料庫中檢索相關的常識知識三元組，然後根據從源上下文到問題的轉換資訊選擇三元組。基于這些資訊性知識三元組，我們設計了兩個輔助任務，一個是概念關系分類，另一個是尾部概念生成。在SQuAD上的實驗結果表明，我們提出的方法能夠顯著提高QG在自動和人工評價名額上的性能，表明将外部常識知識與多任務學習相結合可以幫助模型生成類人的、高品質的問題。
摘要：Question generation (QG) is to generate natural and grammatical questions that can be answered by a specific answer for a given context. Previous sequence-to-sequence models suffer from a problem that asking high-quality questions requires commonsense knowledge as backgrounds, which in most cases can not be learned directly from training data, resulting in unsatisfactory questions deprived of knowledge. In this paper, we propose a multi-task learning framework to introduce commonsense knowledge into question generation process. We first retrieve relevant commonsense knowledge triples from mature databases and select triples with the conversion information from source context to question. Based on these informative knowledge triples, we design two auxiliary tasks to incorporate commonsense knowledge into the main QG model, where one task is Concept Relation Classification and the other is Tail Concept Generation. Experimental results on SQuAD show that our proposed methods are able to noticeably improve the QG performance on both automatic and human evaluation metrics, demonstrating that incorporating external commonsense knowledge with multi-task learning can help the model generate human-like and high-quality questions.
摘要|資訊提取(1篇)
【1】 A Condense-then-Select Strategy for Text Summarization
标題：一種先壓縮後選擇的文本摘要政策
作者：Hou Pong Chan,Irwin King
機構：Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China., Department of Computer and Information Science, University of Macau, Macau SAR
備注：Accepted by Knowledge-Based Systems (KBS) journal
連結：https://arxiv.org/abs/2106.10468
摘要：Select-then-compress是一種流行的混合式文本摘要架構，具有較高的效率。這個架構首先選擇突出的句子，然後獨立地将每個句子濃縮成一個簡明的版本。然而，單獨壓縮句子會忽略文檔的上下文資訊，是以容易删除顯著資訊。針對這一局限性，本文提出了一種新的文本摘要壓縮選擇架構。我們的架構首先同時壓縮每個文檔語句。原始文檔語句及其壓縮版本将成為提取的候選對象。最後，提取器利用文檔的上下文資訊來選擇候選文檔并将它們組合成摘要。如果在壓縮過程中删除了顯著資訊，提取器可以選擇一個原始句子來保留資訊。是以，我們的架構有助于避免顯著資訊的丢失，同時保持句子級壓縮的高效率。在CNN/DailyMail、DUC-2002和Pubmed資料集上的實驗結果表明，我們的架構優于先選擇後壓縮架構和其他強基線。
摘要：Select-then-compress is a popular hybrid, framework for text summarization due to its high efficiency. This framework first selects salient sentences and then independently condenses each of the selected sentences into a concise version. However, compressing sentences separately ignores the context information of the document, and is therefore prone to delete salient information. To address this limitation, we propose a novel condense-then-select framework for text summarization. Our framework first concurrently condenses each document sentence. Original document sentences and their compressed versions then become the candidates for extraction. Finally, an extractor utilizes the context information of the document to select candidates and assembles them into a summary. If salient information is deleted during condensing, the extractor can select an original sentence to retain the information. Thus, our framework helps to avoid the loss of salient information, while preserving the high efficiency of sentence-level compression. Experiment results on the CNN/DailyMail, DUC-2002, and Pubmed datasets demonstrate that our framework outperforms the select-then-compress framework and other strong baselines.
推理|分析|了解|解釋(2篇)
【1】 Understanding the Dynamics between Vaping and Cannabis Legalization Using Twitter Opinions
标題：用推特上的觀點了解大麻合法化和Vaping之間的動态
作者：Shishir Adhikari,Akshay Uppal,Robin Mermelstein,Tanya Berger-Wolf,Elena Zheleva
機構：Computer Science, University of Illinois at Chicago, Psychology; Institute for Health Research and Policy, University of Illinois at Chicago, Computer Science and Engineering; Electrical and Computer Engineering; Evolution, Ecology, and Organismal Biology;
備注：Published at ICWSM 2021
連結：https://arxiv.org/abs/2106.11029
摘要：大麻合法化受到美國許多州的歡迎，但其在從使用煙草電子煙更新為吸食大麻方面的作用尚不清楚。與此同時，吸食大麻與新的肺部疾病和青少年使用率上升有關。為了了解大麻合法化對更新的影響，我們設計了一項觀察性研究，以評估娛樂性大麻合法化對電子煙使用者親大麻态度發展的因果影響。我們收集并分析了Twitter資料，其中包含了對大麻和JUUL（一個非常流行的電子香煙品牌）的看法。我們使用弱監督學習對個人微網誌進行過濾，并分類進行姿态檢測。我們發現，休閑大麻合法化政策對已經支援電子煙的使用者的親大麻态度的發展産生了影響。
摘要：Cannabis legalization has been welcomed by many U.S. states but its role in escalation from tobacco e-cigarette use to cannabis vaping is unclear. Meanwhile, cannabis vaping has been associated with new lung diseases and rising adolescent use. To understand the impact of cannabis legalization on escalation, we design an observational study to estimate the causal effect of recreational cannabis legalization on the development of pro-cannabis attitude for e-cigarette users. We collect and analyze Twitter data which contains opinions about cannabis and JUUL, a very popular e-cigarette brand. We use weakly supervised learning for personal tweet filtering and classification for stance detection. We discover that recreational cannabis legalization policy has an effect on increased development of pro-cannabis attitudes for users already in favor of e-cigarettes.
【2】 Out of Context: A New Clue for Context Modeling of Aspect-based Sentiment Analysis
标題：上下文之外：基于方面情感分析的上下文模組化新線索
作者：Bowen Xing,Ivor W. Tsang
機構：Australian Artificial Intelligence Institute, University of Technology Sydney, Ultimo, NSW , Australia
備注：Submitted to JAIR
連結：https://arxiv.org/abs/2106.10816
摘要：基于方面的情緒分析（ABSA）的目的是預測評論中所表達的關于某一方面的情緒。ABSA的核心是建立上下文與特定方面之間的互動模型，提取與方面相關的資訊。在以往的工作中，人們通常采用注意機制和依賴圖網絡來捕捉上下文和特定方面之間的關系。将上下文隐藏狀态的權重和作為分類器的最終表示。然而，與給定方面相關的資訊可能已經被丢棄，并且不利資訊可能被保留在現有模型的上下文模組化過程中。這一問題是後續子產品無法解決的，原因有二：一是它們的操作都是在編碼器生成的上下文隐藏狀态上進行的，其值在編碼器運作後不能改變；第二，現有的編碼器隻考慮上下文而不考慮給定的方面。為了解決這個問題，我們認為在上下文模組化過程中，給定的方面應該被視為上下文之外的一條新線索。在解決方案方面，我們基于不同的主幹設計了幾個方面感覺的上下文編碼器：一個方面感覺的LSTM和三個方面感覺的BERTs。它們專門用于生成面向方面的隐藏狀态，這些隐藏狀态是為ABSA任務定制的。在這些感覺方面的上下文編碼器中，給定方面的語義被用來調節資訊流。是以，在生成的隐藏狀态中，可以保留與方面相關的資訊，并且可以排除與方面無關的資訊。我們在多個基準資料集上進行了大量的實驗，并進行了實證分析，證明了我們提出的面向方面上下文編碼器的有效性和優越性。
摘要：Aspect-based sentiment analysis (ABSA) aims to predict the sentiment expressed in a review with respect to a given aspect. The core of ABSA is to model the interaction between the context and given aspect to extract the aspect-related information. In prior work, attention mechanisms and dependency graph networks are commonly adopted to capture the relations between the context and given aspect. And the weighted sum of context hidden states is used as the final representation fed to the classifier. However, the information related to the given aspect may be already discarded and adverse information may be retained in the context modeling processes of existing models. This problem cannot be solved by subsequent modules and there are two reasons: first, their operations are conducted on the encoder-generated context hidden states, whose value cannot change after the encoder; second, existing encoders only consider the context while not the given aspect. To address this problem, we argue the given aspect should be considered as a new clue out of context in the context modeling process. As for solutions, we design several aspect-aware context encoders based on different backbones: an aspect-aware LSTM and three aspect-aware BERTs. They are dedicated to generate aspect-aware hidden states which are tailored for ABSA task. In these aspect-aware context encoders, the semantics of the given aspect is used to regulate the information flow. Consequently, the aspect-related information can be retained and aspect-irrelevant information can be excluded in the generated hidden states. We conduct extensive experiments on several benchmark datasets with empirical analysis, demonstrating the efficacies and advantages of our proposed aspect-aware context encoders.
GAN|對抗|攻擊|生成相關(1篇)
【1】 Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training
标題：協同對抗性訓練增強遠端監督關系抽取能力
作者：Tao Chen,Haochen Shi,Liyuan Liu,Siliang Tang,Jian Shao,Zhigang Chen,Yueting Zhuang
機構：Zhejiang University ,University of Illinois at Urbana Champaign ,iFLYTEK Research
備注：Accepted by AAAI 2021
連結：https://arxiv.org/abs/2106.10835
摘要：近年來，随着遠端監督關系提取技術的發展，利用多執行個體學習技術從含噪聲的資料源中提取高品質的監督關系越來越受到人們的重視。在這裡，我們超越了标簽噪聲，發現DS-MIL的關鍵瓶頸是資料使用率低：MIL對高品質的監督進行細化，MIL放棄了大量的訓練執行個體，導緻資料使用率低，阻礙了模型訓練的充分監督。在本文中，我們提出了協同對抗訓練來提高資料使用率，它在不同層次上協調虛拟對抗訓練（VAT）和對抗訓練（AT）。具體來說，因為VAT是無标簽的，是以我們使用執行個體級VAT來回收MIL放棄的執行個體。此外，我們在包級别部署，以充分發揮MIL獲得的高品質監管的潛力。我們所提出的方法帶來了一緻的改進（~5絕對AUC分數）以前的最新狀态，這驗證了資料利用問題的重要性和我們的方法的有效性。
摘要：With recent advances in distantly supervised (DS) relation extraction (RE), considerable attention is attracted to leverage multi-instance learning (MIL) to distill high-quality supervision from the noisy DS. Here, we go beyond label noise and identify the key bottleneck of DS-MIL to be its low data utilization: as high-quality supervision being refined by MIL, MIL abandons a large amount of training instances, which leads to a low data utilization and hinders model training from having abundant supervision. In this paper, we propose collaborative adversarial training to improve the data utilization, which coordinates virtual adversarial training (VAT) and adversarial training (AT) at different levels. Specifically, since VAT is label-free, we employ the instance-level VAT to recycle instances abandoned by MIL. Besides, we deploy AT at the bag-level to unleash the full potential of the high-quality supervision got by MIL. Our proposed method brings consistent improvements (~ 5 absolute AUC score) to the previous state of the art, which verifies the importance of the data utilization issue and the effectiveness of our method.
半/弱/無監督|不确定性(3篇)
【1】 Iterative Network Pruning with Uncertainty Regularization for Lifelong Sentiment Classification
标題：基于不确定性正則化的疊代網絡修剪終生情感分類
作者：Binzong Geng,Min Yang,Fajie Yuan,Shupeng Wang,Xiang Ao,Ruifeng Xu
機構：University of Science and Technology of China, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Westlake University, Key Lab of Intelligent Information Processing of Chinese Academy of Sciences, Institute of Computing Technology, CAS
備注：Accepted by the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2021
連結：https://arxiv.org/abs/2106.11197
摘要：終身學習能力對于情感分類器處理網絡上連續不斷的觀點資訊流至關重要。然而，對于深度神經網絡來說，進行終身學習是非常重要的，因為不斷地訓練增量可用資訊不可避免地會導緻災難性的遺忘或幹擾。本文利用網絡剪枝和權值正則化的原理，提出了一種基于不确定性正則化的疊代網絡剪枝終身情感分類方法。IPRLS通過疊代的方式進行網絡剪枝和不确定性正則化，可以适應單個BERT模型處理來自多個域的連續到達資料，同時避免災難性的遺忘和幹擾。具體地說，我們利用一種疊代剪枝方法來去除大型深度網絡中的備援參數，這樣釋放出來的空間就可以用來學習新的任務，解決災難性的遺忘問題。在學習新任務時，我們沒有保持舊任務不變，而是使用基于貝葉斯線上學習架構的不确定性正則化來限制BERT中舊任務權重的更新，進而實作正後向遷移，也就是說，學習新任務可以提高過去任務的績效，同時保護舊知識不被丢失。此外，我們還提出了一個任務相關的低維殘差函數并行于BERT的每一層，使得IPRLS在學習新任務時不容易丢失基本BERT網絡中的知識。在16個流行評論語料庫上進行的大量實驗表明，所提出的IPRLS方法明顯優于強基線方法。為了再現性，我們将代碼和資料送出至：https://github.com/siat-nlp/IPRLS.
摘要：Lifelong learning capabilities are crucial for sentiment classifiers to process continuous streams of opinioned information on the Web. However, performing lifelong learning is non-trivial for deep neural networks as continually training of incrementally available information inevitably results in catastrophic forgetting or interference. In this paper, we propose a novel iterative network pruning with uncertainty regularization method for lifelong sentiment classification (IPRLS), which leverages the principles of network pruning and weight regularization. By performing network pruning with uncertainty regularization in an iterative manner, IPRLS can adapta single BERT model to work with continuously arriving data from multiple domains while avoiding catastrophic forgetting and interference. Specifically, we leverage an iterative pruning method to remove redundant parameters in large deep networks so that the freed-up space can then be employed to learn new tasks, tackling the catastrophic forgetting problem. Instead of keeping the old-tasks fixed when learning new tasks, we also use an uncertainty regularization based on the Bayesian online learning framework to constrain the update of old tasks weights in BERT, which enables positive backward transfer, i.e. learning new tasks improves performance on past tasks while protecting old knowledge from being lost. In addition, we propose a task-specific low-dimensional residual function in parallel to each layer of BERT, which makes IPRLS less prone to losing the knowledge saved in the base BERT network when learning a new task. Extensive experiments on 16 popular review corpora demonstrate that the proposed IPRLS method sig-nificantly outperforms the strong baselines for lifelong sentiment classification. For reproducibility, we submit the code and data at:https://github.com/siat-nlp/IPRLS.
【2】 ArgFuse: A Weakly-Supervised Framework for Document-Level Event Argument Aggregation
标題：ArgFuse：一種用于文檔級事件參數聚合的弱監督架構
作者：Debanjana Kar,Sudeshna Sarkar,Pawan Goyal
機構：Department of Computer Science & Engineering, Indian Institute of Technology, Kharagpur.
備注：11 pages, 8 figures, Accepted in Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) @ACL-IJCNLP 2021
連結：https://arxiv.org/abs/2106.10862
摘要：大多數現有的資訊提取架構（Wadden et al.，2019；Veyshet等人，2020年）專注于句子級任務，很難從給定文檔中擷取綜合資訊。為了從冗長的文本記錄中生成精确的文檔級資訊架構，我們引入了資訊聚合或參數聚合的任務。更具體地說，我們的目标是過濾不相關的和備援的論據提及，提取在一個句子級和呈現一個文檔級的資訊架構。大多數現有的工作都是為了解決文檔級事件參數提取的相關任務（Yang等人，2018a；Zheng等人，2019a）和使用監督技術的顯著實體識别（Jain等人，2020）。為了從大量的标記資料中去除依賴性，我們探索了使用弱監督技術進行資訊聚合的任務。特别地，我們提出了一種多篩子的抽取算法，該算法采用主動學習政策在低資源環境下高效地工作。為此，我們對自己的測試資料集進行了注釋，其中包含131個文檔資訊架構，并釋出了代碼和資料集，對這一新領域的進一步研究進行了展望。據我們所知，我們是第一個用英語為這項任務建立基線結果的人。我們的資料和代碼在https://github.com/DebanjanaKar/ArgFuse.
摘要：Most of the existing information extraction frameworks (Wadden et al., 2019; Veysehet al., 2020) focus on sentence-level tasks and are hardly able to capture the consolidated information from a given document. In our endeavour to generate precise document-level information frames from lengthy textual records, we introduce the task of Information Aggregation or Argument Aggregation. More specifically, our aim is to filter irrelevant and redundant argument mentions that were extracted at a sentence level and render a document level information frame. Majority of the existing works have been observed to resolve related tasks of document-level event argument extraction (Yang et al., 2018a; Zheng et al., 2019a) and salient entity identification (Jain et al.,2020) using supervised techniques. To remove dependency from large amounts of labelled data, we explore the task of information aggregation using weakly-supervised techniques. In particular, we present an extractive algorithm with multiple sieves which adopts active learning strategies to work efficiently in low-resource settings. For this task, we have annotated our own test dataset comprising of 131 document information frames and have released the code and dataset to further research prospects in this new domain. To the best of our knowledge, we are the first to establish baseline results for this task in English. Our data and code are publicly available at https://github.com/DebanjanaKar/ArgFuse.
【3】 CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction
标題：CIL：用于遠端監督關系抽取的對比執行個體學習架構
作者：Tao Chen,Haizhou Shi,Siliang Tang,Zhigang Chen,Fei Wu,Yueting Zhuang
機構：Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, State Key Laboratory of Cognitive Intelligence, Hefei, China
備注：Accepted by ACL 2021
連結：https://arxiv.org/abs/2106.10855
摘要：自DS首次引入關系抽取（RE）任務以來，遠端監控（DS）生成的訓練資料的降噪之旅已經開始。在過去的十年中，研究人員應用多執行個體學習（MIL）架構從一袋句子中尋找最可靠的特征。雖然MIL-bags模式可以大大降低DS噪聲，但它不能表示資料集中許多有用的句子特征。在很多情況下，這些句子特征隻能通過額外的句子級人工标注來擷取，代價很高。是以，遠端監督重模型的性能是有界的。在本文中，我們超越了典型的MIL架構，提出了一種新的對比執行個體學習（CIL）架構。具體而言，我們将初始MIL視為相關的三重編碼器，并對每個執行個體的正對和負對進行限制。實驗證明了該架構的有效性，并在NYT10、GDS和KBP上進行了改進。
摘要：The journey of reducing noise from distant supervision (DS) generated training data has been started since the DS was first introduced into the relation extraction (RE) task. For the past decade, researchers apply the multi-instance learning (MIL) framework to find the most reliable feature from a bag of sentences. Although the pattern of MIL bags can greatly reduce DS noise, it fails to represent many other useful sentence features in the datasets. In many cases, these sentence features can only be acquired by extra sentence-level human annotation with heavy costs. Therefore, the performance of distantly supervised RE models is bounded. In this paper, we go beyond typical MIL framework and propose a novel contrastive instance learning (CIL) framework. Specifically, we regard the initial MIL as the relational triple encoder and constraint positive pairs against negative pairs for each instance. Experiments demonstrate the effectiveness of our proposed framework, with significant improvements over the previous methods on NYT10, GDS and KBP.
檢測相關(1篇)
【1】 Hybrid approach to detecting symptoms of depression in social media entries
标題：檢測社交媒體條目中抑郁症狀的混合方法
作者：Agnieszka Wołk,Karol Chlasta,Paweł Holas
機構：Polish-Japanese Academy of Information, Technology, The Institute of Literary Research of the, Polish Academy of Sciences, Koszykowa ,-, Warsaw, Nowy Świat ,-, Warsaw, Kozminski University, Jagiellońska ,,-, Warsaw, University of Warsaw
備注：11 pages, 4 figures, 2 tables, The Pacific Asia Conference on Information Systems (PACIS2021)
連結：https://arxiv.org/abs/2106.10485
摘要：情緒和詞彙分析被廣泛用于檢測抑郁症或焦慮症。有文獻記載，與健康人相比，情緒障礙患者所使用的語言存在顯著差異。盡管如此，這些詞彙方法的有效性還可以進一步提高，因為目前的分析重點是社交媒體條目是關于什麼的，而不是它們是如何寫的。在這項研究中，我們将重點放在這些短文彼此相似的方面，以及它們是如何産生的。我們提出了一種新穎的方法來解決抑郁症篩查問題，它是一種已知的從文本中擷取語言資訊的有效方法。我們将這些結果與基于BERT結構的情感分析進行了比較。最後，我們建立了一個混合模型，實作了71%的診斷準确率。
摘要：Sentiment and lexical analyses are widely used to detect depression or anxiety disorders. It has been documented that there are significant differences in the language used by a person with emotional disorders in comparison to a healthy individual. Still, the effectiveness of these lexical approaches could be improved further because the current analysis focuses on what the social media entries are about, and not how they are written. In this study, we focus on aspects in which these short texts are similar to each other, and how they were created. We present an innovative approach to the depression screening problem by applying Collgram analysis, which is a known effective method of obtaining linguistic information from texts. We compare these results with sentiment analysis based on the BERT architecture. Finally, we create a hybrid model achieving a diagnostic accuracy of 71%.
識别/分類(2篇)
【1】 Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification
标題：穩健性是否能提高公平性？基于詞替換穩健性的文本分類公平性研究
作者：Yada Pruksachatkun,Satyapriya Krishna,Jwala Dhamala,Rahul Gupta,Kai-Wei Chang
機構：UCLA, Amazon Alexa
連結：https://arxiv.org/abs/2106.10826
摘要：現有的偏倚緩解方法，以減少不同隊列之間的模型結果的差異，集中在資料增加，減少模型嵌入，或增加公平性為基礎的優化目标，在訓練。另外，已開發出經驗證的詞替換魯棒性方法，以減少虛假特征和同義詞替換對模型預測的影響。雖然它們的最終目标不同，但它們都旨在鼓勵模型對輸入的某些變化做出相同的預測。在這篇論文中，我們研究了驗證詞替換穩健性方法在多文本分類任務中改善機會均等和幾率均等的效用。我們觀察到，經過認證的穩健性方法提高了公平性，并且在訓練中同時使用穩健性和偏差緩解方法在兩個方面都取得了改進
摘要：Existing bias mitigation methods to reduce disparities in model outcomes across cohorts have focused on data augmentation, debiasing model embeddings, or adding fairness-based optimization objectives during training. Separately, certified word substitution robustness methods have been developed to decrease the impact of spurious features and synonym substitutions on model predictions. While their end goals are different, they both aim to encourage models to make the same prediction for certain changes in the input. In this paper, we investigate the utility of certified word substitution robustness methods to improve equality of odds and equality of opportunity on multiple text classification tasks. We observe that certified robustness methods improve fairness, and using both robustness and bias mitigation methods in training results in an improvement in both fronts
【2】 Improving Compositional Generalization in Classification Tasks via Structure Annotations
标題：通過結構标注改進分類任務中的成分泛化
作者：Juyong Kim,Pradeep Ravikumar,Joshua Ainslie,Santiago Ontañón
機構：Carnegie Mellon University, Santiago Onta˜n´on, Google Research
備注：Accepted as a short paper at ACL 2021
連結：https://arxiv.org/abs/2106.10434
摘要：組合泛化是通過組合已知的成分，系統地泛化到一個新的資料分布的能力。雖然人類似乎有很強的合成概括能力，但最先進的神經模型很難做到這一點。在這項工作中，我們研究了分類任務中的合成概括，并提出了兩個主要貢獻。首先，我們研究如何将自然語言序列轉換成序列資料集，再轉換成同樣需要合成泛化的分類資料集。其次，我們展示了提供結構提示（特别是提供解析樹和實體連結作為轉換器模型的注意遮罩）有助于合成泛化。
摘要：Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components. Although humans seem to have a great ability to generalize compositionally, state-of-the-art neural models struggle to do so. In this work, we study compositional generalization in classification tasks and present two main contributions. First, we study ways to convert a natural language sequence-to-sequence dataset to a classification dataset that also requires compositional generalization. Second, we show that providing structural hints (specifically, providing parse trees and entity links as attention masks for a Transformer model) helps compositional generalization.
表征(1篇)
【1】 Do Encoder Representations of Generative Dialogue Models Encode Sufficient Information about the Task ?
标題：生成性對話模型的編碼表示是否編碼了關于任務的足夠資訊？
作者：Prasanna Parthasarathi,Joelle Pineau,Sarath Chandar
機構： School of Computer Science, McGill University, Quebec Artificial Intelligence Institute (Mila), Canada, École Polytechnique de Montréal, Canada CIFAR AI Chair
備注：Accepted at SIGDial 2021. arXiv admin note: substantial text overlap with arXiv:2008.10427
連結：https://arxiv.org/abs/2106.10622
摘要：在資料驅動的方法中，預測對話中的下一個話語取決于對使用者輸入文本的編碼，進而生成适當的相關響應。雖然生成的語言的語義和句法品質得到了評估，但輸入的編碼表示往往沒有得到評估。由于編碼器的表示對于預測适當的響應是必不可少的，是以編碼器表示的評估是一個具有挑戰性的重要問題。在這項工作中，我們展示了評估通過人工或自動度量生成的文本不足以恰當地評估對話模型的語言了解的可靠性，為此，我們提出了一組探測任務來評估對話模型中常用的不同語言編碼器的編碼器表示。通過實驗，我們觀察到有些探測任務更容易，有些甚至對于複雜的模型體系結構來說更難學習。并且，通過實驗我們觀察到基于RNN的體系結構在文本生成的自動度量上比transformer模型的性能要低，但是在探測任務上比transformer模型的性能更好，這表明RNN可能比transformer模型更好地儲存任務資訊。
摘要：Predicting the next utterance in dialogue is contingent on encoding of users' input text to generate appropriate and relevant response in data-driven approaches. Although the semantic and syntactic quality of the language generated is evaluated, more often than not, the encoded representation of input is not evaluated. As the representation of the encoder is essential for predicting the appropriate response, evaluation of encoder representation is a challenging yet important problem. In this work, we showcase evaluating the text generated through human or automatic metrics is not sufficient to appropriately evaluate soundness of the language understanding of dialogue models and, to that end, propose a set of probe tasks to evaluate encoder representation of different language encoders commonly used in dialogue models. From experiments, we observe that some of the probe tasks are easier and some are harder for even sophisticated model architectures to learn. And, through experiments we observe that RNN based architectures have lower performance on automatic metrics on text generation than transformer model but perform better than the transformer model on the probe tasks indicating that RNNs might preserve task information better than the Transformers.
Word2Vec|文本|單詞(1篇)
【1】 Multi-Pair Text Style Transfer on Unbalanced Data
标題：不平衡資料上的多對文本樣式轉換
作者：Xing Han,Jessica Lundin
機構：University of Texas at Austin, Austin, TX, Salesforce, San Francisco, CA
備注：Meta Learning and Its Applications to Natural Language Processing, ACL 2021 Workshop
連結：https://arxiv.org/abs/2106.10608
摘要：文本風格轉換的目的是在不改變文本内容的情況下，通過複述句子或替換關鍵字，将一個領域的文本轉換成另一個領域的文本。在必要的情況下，最先進的方法已經發展到适應非平行的訓練資料，因為通常情況下，有多個大小不等的資料源，有标記和無标記的句子混合在一起。此外，在每個源中定義的固有樣式可能是不同的。通用的雙向（例如，正式的$\Leftrightarrow$非正式的）樣式傳輸，不管不同的組，可能無法很好地推廣到不同的應用程式。在這項工作中，我們開發了一個任務自适應元學習架構，可以同時執行多對文本風格轉換使用一個單一的模式。該方法能夠自适應地平衡多任務間元知識的差異。結果表明，我們的方法可以得到更好的定量性能以及連貫的風格變化。這種方法很好地解決了資料不平衡和域不比對的常見問題。
摘要：Text-style transfer aims to convert text given in one domain into another by paraphrasing the sentence or substituting the keywords without altering the content. By necessity, state-of-the-art methods have evolved to accommodate nonparallel training data, as it is frequently the case there are multiple data sources of unequal size, with a mixture of labeled and unlabeled sentences. Moreover, the inherent style defined within each source might be distinct. A generic bidirectional (e.g., formal $\Leftrightarrow$ informal) style transfer regardless of different groups may not generalize well to different applications. In this work, we developed a task adaptive meta-learning framework that can simultaneously perform a multi-pair text-style transfer using a single model. The proposed method can adaptively balance the difference of meta-knowledge across multiple tasks. Results show that our method leads to better quantitative performance as well as coherent style variations. Common challenges of unbalanced data and mismatched domains are handled well by this method.
其他神經網絡|深度學習|模型|模組化(9篇)
【1】 A Discriminative Entity-Aware Language Model for Virtual Assistants
标題：一種區分實體的虛拟助手語言模型
作者：Mandana Saebi,Ernest Pusateri,Aaksha Meghawat,Christophe Van Gysel
機構：University of Notre Dame, Notre Dame, IN, USA, Apple, Cupertino, CA, USA
備注：To appear in Interspeech 2021
連結：https://arxiv.org/abs/2106.11292
摘要：高品質的自動語音識别（ASR）是虛拟助理（VAs）正常工作的關鍵。但是，ASR在包含命名實體的VA請求上的性能通常很差。在這項工作中，我們從觀察命名實體上的許多ASR錯誤與實際知識不一緻開始。我們使用捕捉實體類型實體和實體-實體關系的特征，擴充了以往的區分性n-gram語言模組化方法，将知識圖（KG）中的真實世界知識結合起來。我們通過一個有效的格重排序過程來應用我們的模型，在一些包含不太流行的實體的綜合測試集上實作了超過25%的相對句子錯誤率降低，而在一個均勻采樣的VA測試集上退化最小。
摘要：High-quality automatic speech recognition (ASR) is essential for virtual assistants (VAs) to work well. However, ASR often performs poorly on VA requests containing named entities. In this work, we start from the observation that many ASR errors on named entities are inconsistent with real-world knowledge. We extend previous discriminative n-gram language modeling approaches to incorporate real-world knowledge from a Knowledge Graph (KG), using features that capture entity type-entity and entity-entity relationships. We apply our model through an efficient lattice rescoring process, achieving relative sentence error rate reductions of more than 25% on some synthesized test sets covering less popular entities, with minimal degradation on a uniformly sampled VA test set.
【2】 Self-Calibrating Neural-Probabilistic Model for Authorship Verification Under Covariate Shift
标題：協變量漂移下的作者自校準神經機率模型
作者：Benedikt Boenninghoff,Dorothea Kolossa,Robert M. Nickel
機構： Ruhr University Bochum, Bucknell University, USA
備注：12th International Conference of the CLEF Association, 2021
連結：https://arxiv.org/abs/2106.11196
摘要：我們正在解決兩個基本問題，在作者身份驗證（AV）：主題變異和校準錯誤。兩個有争議的文本主題的變化是大多數視聽系統出錯的主要原因。此外，可以觀察到，由深度學習AV機制産生的潛在機率估計常常與相應訓練資料中的實際案例計數不比對。是以，機率估計的校準很差。我們正在擴充我們的架構，從泛2020年，包括貝葉斯因子評分（BFS）和不确定性适應層（UAL）來解決這兩個問題。通過對2020/21pan-AV共享任務資料的實驗表明，該方法顯著降低了對局部變化的敏感性，顯著提高了系統的标定精度。
摘要：We are addressing two fundamental problems in authorship verification (AV): Topic variability and miscalibration. Variations in the topic of two disputed texts are a major cause of error for most AV systems. In addition, it is observed that the underlying probability estimates produced by deep learning AV mechanisms oftentimes do not match the actual case counts in the respective training data. As such, probability estimates are poorly calibrated. We are expanding our framework from PAN 2020 to include Bayes factor scoring (BFS) and an uncertainty adaptation layer (UAL) to address both problems. Experiments with the 2020/21 PAN AV shared task data show that the proposed method significantly reduces sensitivities to topical variations and significantly improves the system's calibration.
【3】 Explicit Interaction Network for Aspect Sentiment Triplet Extraction
标題：面向體感三元組提取的顯式互動網絡
作者：Peiyi Wang,Lianzhe Huang,Tianyu Liu,Damai Dai,Runxin Xu,Houfeng Wang,Baobao Chang,Zhifang Sui
機構：MOE Key Lab of Computational Linguistics, Peking University, Beijing, China
連結：https://arxiv.org/abs/2106.11148
摘要：體感三元組提取（ASTE）的目的是識别目标、目标的情感極性和解釋句子情感的觀點。ASTE可以自然地劃分為3個子任務，即目标檢測、觀點檢測和情感分類。我們認為正确的子任務組合、目标意見對的組合特征提取以及子任務之間的互動是成功的關鍵。然而，以前的工作可能會在“一對多”或“多對一”的情況下失敗，或者由于子任務的表述有缺陷、次優的特征表示或缺乏子任務互動而産生不存在的情感三胞胎。本文将ASTE分為符合人類認知的目标觀點聯合檢測和情感分類兩個子任務，并相應地提出了序列編碼器和表編碼器。表編碼器在标記對層次上提取情感資訊，進而可以很容易地擷取目标和觀點之間的組合特征。為了建立子任務之間的顯式互動，我們利用表表示來指導序列編碼，并将序列特征注入到表編碼器中。實驗表明，在6個流行的ASTE資料集上，我們的模型優于現有的方法。
摘要：Aspect Sentiment Triplet Extraction (ASTE) aims to recognize targets, their sentiment polarities and opinions explaining the sentiment from a sentence. ASTE could be naturally divided into 3 atom subtasks, namely target detection, opinion detection and sentiment classification. We argue that the proper subtask combination, compositional feature extraction for target-opinion pairs, and interaction between subtasks would be the key to success. Prior work, however, may fail on `one-to-many' or `many-to-one' situations, or derive non-existent sentiment triplets due to defective subtask formulation, sub-optimal feature representation or the lack of subtask interaction. In this paper, we divide ASTE into target-opinion joint detection and sentiment classification subtasks, which is in line with human cognition, and correspondingly propose sequence encoder and table encoder. Table encoder extracts sentiment at token-pair level, so that the compositional feature between targets and opinions can be easily captured. To establish explicit interaction between subtasks, we utilize the table representation to guide the sequence encoding, and inject the sequence features back into the table encoder. Experiments show that our model outperforms state-of-the-art methods on six popular ASTE datasets.
【4】 Leveraging Language to Learn Program Abstractions and Search Heuristics
标題：利用語言學習程式抽象和搜尋啟發式
作者：Catherine Wong,Kevin Ellis,Joshua B. Tenenbaum,Jacob Andreas
機構： a framework for improving the ef-ficiency and generalizability of learned program synthesis 1MIT 2Cornell University 3Center for Brains
備注：appeared in Thirty-eighth International Conference on Machine Learning (ICML 2021)
連結：https://arxiv.org/abs/2106.11053
摘要：歸納程式綜合，或從期望行為的例子中推斷程式，為建立可解釋的、健壯的和可推廣的機器學習系統提供了一個通用的範例。有效的程式綜合取決于兩個關鍵要素：一個強大的函數庫，從中生成程式；一個高效的搜尋政策，用于查找解決給定任務的程式。我們介紹了LAPS（Language for Abstraction and Program Search），一種使用自然語言注釋來指導庫的聯合學習的技術，以及用于合成的神經引導搜尋模型。當內建到最先進的庫學習系統（DreamCoder）中時，LAPS生成更高品質的庫，并在字元串編輯、圖像合成和場景抽象推理三個領域提高搜尋效率和泛化能力，即使在測試時沒有可用的自然語言提示。
摘要：Inductive program synthesis, or inferring programs from examples of desired behavior, offers a general paradigm for building interpretable, robust, and generalizable machine learning systems. Effective program synthesis depends on two key ingredients: a strong library of functions from which to build programs, and an efficient search strategy for finding programs that solve a given task. We introduce LAPS (Language for Abstraction and Program Search), a technique for using natural language annotations to guide joint learning of libraries and neurally-guided search models for synthesis. When integrated into a state-of-the-art library learning system (DreamCoder), LAPS produces higher-quality libraries and improves search efficiency and generalization on three domains -- string editing, image composition, and abstract reasoning about scenes -- even when no natural language hints are available at test time.
【5】 Interventional Video Grounding with Dual Contrastive Learning
标題：基于雙重對比學習的介入性視訊尋根
作者：Guoshun Nan,Rui Qiao,Yao Xiao,Jun Liu,Sicong Leng,Hao Zhang,Wei Lu
機構： StatNLP Research Group, Singapore University of Technology and Design, Shanghai Jiao Tong University, China, Information Systems Technology and Design, Singapore University of Technology and Design, Singapore
備注：Accepted in CVPR 2021
連結：https://arxiv.org/abs/2106.11013
摘要：視訊接地的目的是定位一個時刻從一個未經剪輯的視訊為一個給定的文本查詢。現有的方法更多地關注視覺和語言刺激與各種基于可能性的比對或回歸政策的比對，即P（Y | X）。是以，由于資料集的選擇偏差，這些模型可能會受到語言和視訊特征之間虛假相關性的影響。1） 為了揭示模型和資料背後的因果關系，我們首先從因果推理的角度提出了一種新的範式，即基于結構化因果模型（SCM）和do演算P（Y | do（X））的介入性視訊接地（IVG）。然後，我們提出了一個簡單而有效的方法來近似未觀察到的混雜因素，因為它不能直接從資料集中采樣。2） 同時，我們引入了一種雙重對比學習方法（DCL），通過最大化查詢和視訊片段之間的互資訊（MI）以及目标時刻的開始幀/結束幀與視訊中其他幀之間的互資訊（MI）來更好地對齊文本和視訊，以學習更多資訊的視覺表示。在三個标準基準上的實驗表明了該方法的有效性。
摘要：Video grounding aims to localize a moment from an untrimmed video for a given textual query. Existing approaches focus more on the alignment of visual and language stimuli with various likelihood-based matching or regression strategies, i.e., P(Y|X). Consequently, these models may suffer from spurious correlations between the language and video features due to the selection bias of the dataset. 1) To uncover the causality behind the model and data, we first propose a novel paradigm from the perspective of the causal inference, i.e., interventional video grounding (IVG) that leverages backdoor adjustment to deconfound the selection bias based on structured causal model (SCM) and do-calculus P(Y|do(X)). Then, we present a simple yet effective method to approximate the unobserved confounder as it cannot be directly sampled from the dataset. 2) Meanwhile, we introduce a dual contrastive learning approach (DCL) to better align the text and video by maximizing the mutual information (MI) between query and video clips, and the MI between start/end frames of a target moment and the others within a video to learn more informative visual representations. Experiments on three standard benchmarks show the effectiveness of our approaches.
【6】 TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning
标題：TCIC：主題概念、跨語言學習和圖像字幕視覺
作者：Zhihao Fan,Zhongyu Wei,Siyuan Wang,Ruize Wang,Zejun Li,Haijun Shan,Xuanjing Huang
機構：Zhejiang Lab, Research Institute of Intelligent and Complex Systems, Fudan University, China
備注：IJCAI2021
連結：https://arxiv.org/abs/2106.10936
摘要：現有的圖像字幕的研究通常是用一個具有低層次事實（對象和關系）的場景圖來表示圖像，而沒有捕捉到高層次的語義。在本文中，我們提出了一個主題概念擴充圖像字幕（TCIC）架構，其中包含了主題概念來表示進階跨模态語義。在實踐中，我們将主題概念模組化為記憶向量，并提出了主題節點轉換器（Transformer-with-theme-Nodes，TTN）來整合這些向量用于圖像字幕。考慮到主題概念可以從圖像和字幕中學習，我們提出了兩種基于TTN的主題概念表征學習設定。在視覺方面，TTN被配置為将基于場景圖的特征和主題概念作為視覺表示學習的輸入。在語言方面，TTN被配置為将字幕和主題概念作為文本表示重構的輸入。這兩種設定都旨在使用相同的基于轉換器的解碼器生成目标字幕。在訓練過程中，我們進一步将從圖像中學習到的主題概念的表達與相應的字幕對齊，以加強跨模态學習。在MS-COCO上的實驗結果表明，與一些最新的模型相比，我們的方法是有效的。
摘要：Existing research for image captioning usually represents an image using a scene graph with low-level facts (objects and relations) and fails to capture the high-level semantics. In this paper, we propose a Theme Concepts extended Image Captioning (TCIC) framework that incorporates theme concepts to represent high-level cross-modality semantics. In practice, we model theme concepts as memory vectors and propose Transformer with Theme Nodes (TTN) to incorporate those vectors for image captioning. Considering that theme concepts can be learned from both images and captions, we propose two settings for their representations learning based on TTN. On the vision side, TTN is configured to take both scene graph based features and theme concepts as input for visual representation learning. On the language side, TTN is configured to take both captions and theme concepts as input for text representation re-construction. Both settings aim to generate target captions with the same transformer-based decoder. During the training, we further align representations of theme concepts learned from images and corresponding captions to enforce the cross-modality learning. Experimental results on MS COCO show the effectiveness of our approach compared to some state-of-the-art models.
【7】 Context-Aware Legal Citation Recommendation using Deep Learning
标題：基于深度學習的上下文感覺法律引文推薦
作者：Zihan Huang,Charles Low,Mengqiu Teng,Hongyi Zhang,Daniel E. Ho,Mark S. Krass,Matthias Grabmair
機構：Language Technologies Institute, Carnegie Mellon University, Stanford University, Department of Informatics, Technical University of Munich, SINC GmbH
備注：10 pages published in Proceedings of ICAIL 2021; link to data here: this https URL ; code available here: this https URL
連結：https://arxiv.org/abs/2106.10776
摘要：律師和法官花了大量時間研究起草判決時引用的适當法律權威。在本文中，我們開發了一個引文推薦工具，可以幫助提高意見起草過程中的效率。我們訓練了四種機器學習模型，包括一種基于引文清單的方法（協作過濾）和三種基于上下文的方法（文本相似度、BiLSTM和RoBERTa分類器）。我們的實驗表明，利用本地文本上下文可以提高推薦，并且深層神經模型可以獲得良好的性能。我們表明，非深度文本方法受益于對結構化案例中繼資料的通路，但深度模型僅在從長度不足的上下文進行預測時受益于這種通路。我們還發現，即使經過廣泛的訓練，RoBERTa并沒有超過一個循環神經模型，盡管它的好處是預訓練。我們對RoBERTa模型的行為分析進一步表明，預測性能在時間和引文類别上是穩定的。
摘要：Lawyers and judges spend a large amount of time researching the proper legal authority to cite while drafting decisions. In this paper, we develop a citation recommendation tool that can help improve efficiency in the process of opinion drafting. We train four types of machine learning models, including a citation-list based method (collaborative filtering) and three context-based methods (text similarity, BiLSTM and RoBERTa classifiers). Our experiments show that leveraging local textual context improves recommendation, and that deep neural models achieve decent performance. We show that non-deep text-based methods benefit from access to structured case metadata, but deep models only benefit from such access when predicting from context of insufficient length. We also find that, even after extensive training, RoBERTa does not outperform a recurrent neural model, despite its benefits of pretraining. Our behavior analysis of the RoBERTa model further shows that predictive performance is stable across time and citation classes.
【8】 CPM-2: Large-scale Cost-effective Pre-trained Language Models
标題：CPM-2：大規模高成本效益的預訓練語言模型
作者：Zhengyan Zhang,Yuxian Gu,Xu Han,Shengqi Chen,Chaojun Xiao,Zhenbo Sun,Yuan Yao,Fanchao Qi,Jian Guan,Pei Ke,Yanzheng Cai,Guoyang Zeng,Zhixing Tan,Zhiyuan Liu,Minlie Huang,Wentao Han,Yang Liu,Xiaoyan Zhu,Maosong Sun
機構：Department of Computer Science and Technology, Tsinghua University & BAAI
連結：https://arxiv.org/abs/2106.10715
摘要：近年來，預訓練語言模型（plm）的規模突飛猛進。然而，這些大規模plm的效率問題限制了它們在現實場景中的應用。我們提出了一套成本效益的技術，使用PLM來處理預訓練、微調和推理的效率問題(1） 我們引入知識繼承來加速預訓練過程，利用現有的plm代替從頭開始的訓練模型(2） 我們探索用大規模鎖相環進行快速調諧的最佳實踐。與正常微調相比，快速微調顯著減少了任務特定參數的數量(3） 我們實作了一個新的推理工具InfMoE，用于在有限的計算資源下使用大規模plm。基于我們的高成本效益流水線，我們預先訓練了兩個模型：110億參數的編譯碼雙語模型（CPM-2）和1980億參數的MoE模型。在我們的實驗中，我們比較了下遊任務的CPM-2和mT5。實驗結果表明，CPM-2具有良好的通用語言智能。此外，我們還驗證了InfMoE在單個GPU上對具有數百億個參數的大規模模型進行推理時的有效性。所有源代碼和模型參數都可以在https://github.com/TsinghuaAI/CPM.
摘要：In recent years, the size of pre-trained language models (PLMs) has grown by leaps and bounds. However, efficiency issues of these large-scale PLMs limit their utilization in real-world scenarios. We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference. (1) We introduce knowledge inheritance to accelerate the pre-training process by exploiting existing PLMs instead of training models from scratch. (2) We explore the best practice of prompt tuning with large-scale PLMs. Compared with conventional fine-tuning, prompt tuning significantly reduces the number of task-specific parameters. (3) We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs with limited computational resources. Based on our cost-effective pipeline, we pre-train two models: an encoder-decoder bilingual model with 11 billion parameters (CPM-2) and its corresponding MoE version with 198 billion parameters. In our experiments, we compare CPM-2 with mT5 on downstream tasks. Experimental results show that CPM-2 has excellent general language intelligence. Moreover, we validate the efficiency of InfMoE when conducting inference of large-scale models having tens of billions of parameters on a single GPU. All source code and model parameters are available at https://github.com/TsinghuaAI/CPM.
【9】 Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
标題：使用以值為目标的資料集使語言模型适應社會(Palms)的過程
作者：Irene Solaiman,Christy Dennison
機構：OpenAI
備注：Both authors contributed equally. Submitted to NeurIPS 2021
連結：https://arxiv.org/abs/2106.10328
摘要：語言模型會産生有害的和有偏見的輸出，并表現出不受歡迎的行為。我們提出了一個用以值為目标的資料集使語言模型适應社會（PALMS）的過程，這是一個疊代過程，通過對反映一組預定目标值的資料集進行精心設計和微調來顯著改變模型行為。我們使用三個名額來評估我們的過程：定量名額與人類評估，評分輸出遵守目标值，和毒性評分的産出；定性名額分析與特定社會類别相關的最常見詞彙。通過每次疊代，我們添加額外的訓練資料集的例子，根據觀察到的缺點，從評估。與基線模型和控制模型相比，PALMS在各種GPT-3語言模型尺寸上的性能都有顯著的提高，而且不影響功能完整性。我們發現手掌的有效性随着模型尺寸的增加而增加。我們證明了在一個小的、手工管理的資料集上顯著調整語言模型行為是可行的。
摘要：Language models can generate harmful and biased outputs and exhibit undesirable behavior. We propose a Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets, an iterative process to significantly change model behavior by crafting and fine-tuning on a dataset that reflects a predetermined set of target values. We evaluate our process using three metrics: quantitative metrics with human evaluations that score output adherence to a target value, and toxicity scoring on outputs; and qualitative metrics analyzing the most common word associated with a given social category. Through each iteration, we add additional training dataset examples based on observed shortcomings from evaluations. PALMS performs significantly better on all metrics compared to baseline and control models for a broad range of GPT-3 language model sizes without compromising capability integrity. We find that the effectiveness of PALMS increases with model size. We show that significantly adjusting language model behavior is feasible with a small, hand-curated dataset.
其他(7篇)
【1】 Abstract Geometrical Computation 11: Slanted Firing Squad Synchronisation on Signal Machines
标題：抽象幾何計算11：信号機上的斜射方隊同步
作者：Jérôme Durand-Lose,Aurélien Emmanuel
機構：Université d’Orléans, INSA Centre Val de Loire, LIFO EA , FR-, Orléans, France
備注：21 pages,29 figures
連結：https://arxiv.org/abs/2106.11176
摘要：元胞自動機上的行刑隊同步是有限多個元胞在沒有任何先驗知識的情況下動态同步。這可以被認為是一個具有無限速度的信号。大多數提議的構造自然地轉化為信号機的連續設定，并生成在水準線上累積的分形圖形，即在時空圖中同步地。在《抽象幾何計算》系列文章中對信号機進行了研究。在本文中，我們設計了一個能夠在任何非無限斜坡上同步/積累的信号機。斜率在初始配置中進行編碼。這是通過構造一個無限樹來實作的，這樣每個節點都計算樹擴充的方式。抽象幾何計算的興趣在于擺脫離散空間的限制，同時解決連續空間的新困難。本文的目的是為進一步研究信号機模型中的可計算累積線提供基本工具。
摘要：Firing Squad Synchronisation on Cellular Automata is the dynamical synchronisation of finitely many cells without any prior knowledge of their range. This can be conceived as a signal with an infinite speed. Most of the proposed constructions naturally translate to the continuous setting of signal machines and generate fractal figures with an accumulation on a horizontal line, i.e. synchronously, in the space-time diagram. Signal machines are studied in a series of articles named Abstract Geometrical Computation. In the present article, we design a signal machine that is able to synchronise/accumulate on any non-infinite slope. The slope is encoded in the initial configuration. This is done by constructing an infinite tree such that each node computes the way the tree expands. The interest of Abstract Geometrical computation is to do away with the constraint of discrete space, while tackling new difficulties from continuous space. The interest of this paper in particular is to provide basic tools for further study of computable accumulation lines in the signal machine model.
【2】 QuaPy: A Python-Based Framework for Quantification
标題：QuaPy：一個基于Python的量化架構
作者：Alejandro Moreo,Andrea Esuli,Fabrizio Sebastiani
機構：Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Via Giuseppe Moruzzi , Pisa, Italy
連結：https://arxiv.org/abs/2106.11057
摘要：QuaPy是一個用Python編寫的用于執行量化（也稱為監督流行率估計）的開源架構。量化是通過監督學習訓練量詞的任務，其中量詞是一個預測因子，用于估計未标記資料樣本中感興趣類别的相對頻率（又稱流行值）。雖然可以通過對每個未标記的資料項應用标準分類器并計算配置設定給每個類的資料項的數量來執行量化，但是已經表明，這種“分類和計數”方法的性能優于專門為量化設計的方法。QuaPy提供了許多基線方法和進階量化方法的實作、面向量化的模型選擇例程、一些廣泛接受的評估度量以及在現場正常使用的健壯的評估協定。QuaPy還提供了常用于測試量詞的資料集，并提供了便于分析和解釋結果的可視化工具。該軟體是開放源代碼的，通過BSD-3許可證公開提供https://github.com/HLT-ISTI/QuaPy，并可通過pip安裝(https://pypi.org/project/QuaPy/)
摘要：QuaPy is an open-source framework for performing quantification (a.k.a. supervised prevalence estimation), written in Python. Quantification is the task of training quantifiers via supervised learning, where a quantifier is a predictor that estimates the relative frequencies (a.k.a. prevalence values) of the classes of interest in a sample of unlabelled data. While quantification can be trivially performed by applying a standard classifier to each unlabelled data item and counting how many data items have been assigned to each class, it has been shown that this "classify and count" method is outperformed by methods specifically designed for quantification. QuaPy provides implementations of a number of baseline methods and advanced quantification methods, of routines for quantification-oriented model selection, of several broadly accepted evaluation measures, and of robust evaluation protocols routinely used in the field. QuaPy also makes available datasets commonly used for testing quantifiers, and offers visualization tools for facilitating the analysis and interpretation of the results. The software is open-source and publicly available under a BSD-3 licence via https://github.com/HLT-ISTI/QuaPy, and can be installed via pip (https://pypi.org/project/QuaPy/)
【3】 Conversational Agents in Software Engineering: Survey, Taxonomy and Challenges
标題：軟體工程中的會話代理：綜述、分類和挑戰
作者：Quim Motger,Xavier Franch,Jordi Marco
機構：(ESSI), Universitat Politècnica de Catalunya (UPC), Spain
備注：37 pages, 15 figures, 2 tables, submitted to journal
連結：https://arxiv.org/abs/2106.10901
摘要：通過專門的科學和工業研究，自然語言接口在人機互動領域的使用正在進行深入的研究。該領域的最新貢獻，包括遞歸神經網絡等深度學習方法、上下文感覺政策的潛力和以使用者為中心的設計方法，使社群重新關注基于軟體的對話系統，通常稱為會話代理或聊天機器人。然而，鑒于該領域的新穎性，缺乏一個通用的、與語境無關的、涵蓋所有研究視角的會話主體研究現狀綜述。在此背景下，本文通過對二次研究的系統文獻回顧，對會話主體的研究現狀進行了綜述。所進行的研究旨在通過在不同領域、研究重點和背景下清晰地呈現最近文獻所發表的聚合知識，形成一個詳盡的視角。是以，本研究提出了會話主體領域不同次元的整體分類，以期對研究者有所幫助，并為未來自然語言界面領域的研究奠定基礎。
摘要：The use of natural language interfaces in the field of human-computer interaction is undergoing intense study through dedicated scientific and industrial research. The latest contributions in the field, including deep learning approaches like recurrent neural networks, the potential of context-aware strategies and user-centred design approaches, have brought back the attention of the community to software-based dialogue systems, generally known as conversational agents or chatbots. Nonetheless, and given the novelty of the field, a generic, context-independent overview on the current state of research of conversational agents covering all research perspectives involved is missing. Motivated by this context, this paper reports a survey of the current state of research of conversational agents through a systematic literature review of secondary studies. The conducted research is designed to develop an exhaustive perspective through a clear presentation of the aggregated knowledge published by recent literature within a variety of domains, research focuses and contexts. As a result, this research proposes a holistic taxonomy of the different dimensions involved in the conversational agents' field, which is expected to help researchers and to lay the groundwork for future research in the field of natural language interfaces.
【4】 Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
标題：注意：多語言、多領域序列模組化中的中心選擇
作者：Hongyu Gong,Yun Tang,Juan Pino,Xian Li
機構：Facebook AI Research
連結：https://arxiv.org/abs/2106.10840
摘要：多頭注意使每個注意頭從輸入序列的不同部分收集顯著資訊，使其成為序列模組化的有力機制。多語言和多領域學習是序列模組化的常見場景，其中的關鍵挑戰是最大化跨語言和領域的正遷移和負遷移。在本文中，我們發現非選擇性的注意力分享是次優的，以實作良好的泛化跨所有語言和領域。我們進一步提出注意共享政策，以促進多語種和多領域序列模組化中的參數共享和專業化。我們的方法自動學習不同語言和領域的共享和專門注意頭，以減輕它們的幹擾。在包括語音識别、文本到文本和語音到文本翻譯在内的各種任務中，所提出的注意共享政策始終為建立在多頭注意基礎上的序列模型帶來收益。對于語音到文本的翻譯，我們的方法在多語言環境下，平均産生$13$語言方向上的$2.0$BLEU，在多域環境下，平均産生$3$域上的$2.0$BLEU。
摘要：Multi-head attention has each of the attention heads collect salient information from different parts of an input sequence, making it a powerful mechanism for sequence modeling. Multilingual and multi-domain learning are common scenarios for sequence modeling, where the key challenge is to maximize positive transfer and mitigate negative transfer across languages and domains. In this paper, we find that non-selective attention sharing is sub-optimal for achieving good generalization across all languages and domains. We further propose attention sharing strategies to facilitate parameter sharing and specialization in multilingual and multi-domain sequence modeling. Our approach automatically learns shared and specialized attention heads for different languages and domains to mitigate their interference. Evaluated in various tasks including speech recognition, text-to-text and speech-to-text translation, the proposed attention sharing strategies consistently bring gains to sequence models built upon multi-head attention. For speech-to-text translation, our approach yields an average of $+2.0$ BLEU over $13$ language directions in multilingual setting and $+2.0$ BLEU over $3$ domains in multi-domain setting.
【5】 Calliar: An Online Handwritten Dataset for Arabic Calligraphy
标題：Calliar：一個用于阿拉伯書法的線上手寫資料集
作者：Zaid Alyafeai,Maged S. Al-shaibani,Mustafa Ghaleb,Yousif Ahmed Al-Wajih
機構：Department of Computer Science, King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia , Interdisciplinary Research Center, for Intelligent Secure Systems, Department of Systems Engineering
連結：https://arxiv.org/abs/2106.10745
摘要：書法是阿拉伯文化遺産的重要組成部分。過去它被用來裝飾房屋和清真寺。通常，這樣的書法是由有審美眼光的專家手工設計的。在過去的幾年裡，有相當大的努力，以數字化這類藝術，要麼采取的照片裝飾建築物或繪制他們使用數字裝置。後者被認為是一種線上形式，通過在螢幕上記錄裝置的運動（例如電子筆）來跟蹤繪圖。在文獻中，有許多離線資料集收集了各種各樣的阿拉伯風格的書法。然而，沒有阿拉伯書法的線上資料集。在本文中，我們說明了我們的方法收集和注釋的線上資料集為阿拉伯語書法稱為Calliar，其中包括2500句話。Calliar是為筆劃、字元、單詞和句子級别的預測而注釋的。
摘要：Calligraphy is an essential part of the Arabic heritage and culture. It has been used in the past for the decoration of houses and mosques. Usually, such calligraphy is designed manually by experts with aesthetic insights. In the past few years, there has been a considerable effort to digitize such type of art by either taking a photo of decorated buildings or drawing them using digital devices. The latter is considered an online form where the drawing is tracked by recording the apparatus movement, an electronic pen for instance, on a screen. In the literature, there are many offline datasets collected with a diversity of Arabic styles for calligraphy. However, there is no available online dataset for Arabic calligraphy. In this paper, we illustrate our approach for the collection and annotation of an online dataset for Arabic calligraphy called Calliar that consists of 2,500 sentences. Calliar is annotated for stroke, character, word and sentence level prediction.
【6】 TweeNLP: A Twitter Exploration Portal for Natural Language Processing
标題：TweNLP：一個面向自然語言處理的Twitter探索門戶
作者：Viraj Shah,Shruti Singh,Mayank Singh
機構：Indian Institute of Technology Gandhinagar, Gujarat, India
備注：ACL-IJCNLP Demo Track 2021
連結：https://arxiv.org/abs/2106.10512
摘要：我們介紹tweenp，一個一站式門戶網站，它組織Twitter的自然語言處理（NLP）資料，并建構一個可視化和探索平台。它策劃了19395條推特（截至2021年4月），來自各種NLP會議和一般NLP讨論。它支援多種功能，如TweetExplorer，可以按主題浏覽tweet，在整個會議的組織周期中可視化Twitter活動的見解，發現流行的研究論文和研究人員。它還建立了會議和研讨會送出截止日期的時間表。我們設想tweenp通過将研究論文的tweet與NLPExplorer科學文獻搜尋引擎相結合，成為NLP社群的集體記憶單元。目前系統位于http://nlpexplorer.org/twitter/CFP .
摘要：We present TweeNLP, a one-stop portal that organizes Twitter's natural language processing (NLP) data and builds a visualization and exploration platform. It curates 19,395 tweets (as of April 2021) from various NLP conferences and general NLP discussions. It supports multiple features such as TweetExplorer to explore tweets by topics, visualize insights from Twitter activity throughout the organization cycle of conferences, discover popular research papers and researchers. It also builds a timeline of conference and workshop submission deadlines. We envision TweeNLP to function as a collective memory unit for the NLP community by integrating the tweets pertaining to research papers with the NLPExplorer scientific literature search engine. The current system is hosted at http://nlpexplorer.org/twitter/CFP .
【7】 Non-native English lexicon creation for bilingual speech synthesis
标題：面向雙語語音合成的非母語英語詞彙創設
作者：Arun Baby,Pranav Jawale,Saranya Vinnaitherthan,Sumukh Badam,Nagaraj Adiga,Sharath Adavanne
機構：Zapr Media Labs (Red Brick Lane Marketing Solutions Pvt. Ltd.), India
備注：Accepted for Presentation at Speech Synthesis Workshop (SSW), 2021 (August 2021)
連結：https://arxiv.org/abs/2106.10870
摘要：說英語的雙語者把英語作為他們的語言之一。他們的英語是非母語的，他們的對話是代碼混合的方式。雙國文語轉換（TTS）系統對于非英語母語者的可了解性取決于一個能夠捕捉非英語母語者使用的音位序列的詞彙。然而，由于缺乏非母語英語詞彙，現有的雙語TTS系統除了使用母語詞彙外，還使用了廣泛使用的母語英語詞彙。由于語音中的非母語英語發音與文本中的母語英語詞彙不一緻，在這種TTS系統中合成語音的可懂度大大降低。本文的出發點在于說話人的母語對非母語英語發音的影響。我們提出了一種基于字母-音素對齊的規則擷取方法，将英語本族語詞彙映射到非本族語詞彙。這種映射的有效性是通過比較雙語（印度英語和印地語）TTS系統訓練與不建議的規則。主觀評價結果表明，采用本文提出的非母語英語詞彙規則訓練的雙語TTS系統在偏好上獲得了6%的絕對提高。
摘要：Bilingual English speakers speak English as one of their languages. Their English is of a non-native kind, and their conversations are of a code-mixed fashion. The intelligibility of a bilingual text-to-speech (TTS) system for such non-native English speakers depends on a lexicon that captures the phoneme sequence used by non-native speakers. However, due to the lack of non-native English lexicon, existing bilingual TTS systems employ native English lexicons that are widely available, in addition to their native language lexicon. Due to the inconsistency between the non-native English pronunciation in the audio and native English lexicon in the text, the intelligibility of synthesized speech in such TTS systems is significantly reduced. This paper is motivated by the knowledge that the native language of the speaker highly influences non-native English pronunciation. We propose a generic approach to obtain rules based on letter to phoneme alignment to map native English lexicon to their non-native version. The effectiveness of such mapping is studied by comparing bilingual (Indian English and Hindi) TTS systems trained with and without the proposed rules. The subjective evaluation shows that the bilingual TTS system trained with the proposed non-native English lexicon rules obtains a 6% absolute improvement in preference.