Dialogue systems, also known as conversational agents, are intelligent computer systems which converse with a human via natural language (text, or speech). They generally fall into two categories based on their applications, that is, chit-chat dialogue systems (chatbots) and task-oriented dialogue systems. The former is primarily interacting with humans to provide reasonable responses and entertainment on open domains, whereas the latter aims to help users complete a dedicated task on one specific domain, for example, inquiring about weather, reserving restaurants, or booking flights. This thesis focuses primarily on task-oriented dialogue systems.

The earliest dialogue systems were highly dependent on complicated and hand-crafted rules and logics, which were costly and unscalable. With the recently unprecedented progress in natural language processing propelled by deep learning, statistical dialogue systems have gradually become well used for reducing cost and providing robustness. Traditionally, these systems were highly modularized in a pipelined manner, including spoken language understanding, dialogue state tracking, dialogue management, and response generation. However, modularized design typically requires expensive human labeling and easily leads to error propagation due to module dependencies. On the other hand, end-to-end neural-based models learn the dialogue hidden representation automatically and generate system responses directly, thereby requiring much less human effort and being more scalable to new domains. Although recurrent neural network (RNN) based neural models show promising results, they still suffer from several major drawbacks. First, dominant RNNs are inherently unstable over long-time sequences as RNNs tend to focus more on short-term memories and forcefully compress historical records into one hidden state vector \citep{weston2014memory}. Second, RNNs focus primarily on modeling sequential dependencies, and thus rich graph structure information hidden in the dialogue context is completely ignored. Lastly, effectively incorporating external knowledge into end-to-end task-oriented dialog systems still remains a challenge.

In this thesis, we are dedicated to addressing these limitations of conventional neural models and propose end-to-end learning frameworks to model long-term dialogue context and to learn dialogue graph structures. In addressing the weakness of RNNs processing long sequences, we propose a novel approach to model long-term slot context and to fully utilize the semantic correlation between slots and intents. We adopt a key-value memory network to model slot context dynamically and to track more important slot tags decoded before, which are then fed into our decoder for slot tagging. Furthermore, gated memory information is utilized to perform intent detection, mutually improving both tasks through global optimization. We empirically show that our key-value memory networks enable effectively tracking long-term dialogue context and enhance the performance of spoken language understanding by a large margin.

In addressing modeling the rich graph structures in dialogue utterances, we introduce a new Graph Convolutional LSTM (GC-LSTM) to learn the semantics contained in the graph-structured dialogues by incorporating a powerful graph convolutional operator. Our proposed GC-LSTM can not only capture the spatio-temporal semantic features in a dialogue, but also learn the co-occurrence relationship between intent detection and slot filling. Furthermore, we propose a Graph-to-Sequence learning framework to push the performance of spoken language understanding to a new start-of-the-art.

In addressing effectively integrating knowledge into dialogue generation systems, we propose a novel end-to-end learning framework to incorporate an external knowledge base (KB) and to capture the intrinsic graph semantics of the dialog history. We propose a novel Graph Memory Network (GMN) based sequence-to-sequence model, GraphMemDialogue, to effectively learn the inherent structural information hidden in dialog history, and to model the dynamic interaction between dialog history and KBs. We adopt a modified graph attention network to learn the rich structure representation in the dialog history, whereas the context-aware representation of KB entities are learnt by our novel GMN. To fully exploit this dynamic interaction, we design a learnable memory controller coupled with external KB entity memories to recurrently incorporate dialog history context into KB entities through a multi-hop reasoning mechanism. Experiments on three public datasets show that our GraphMemDialog model achieves state-of-the-art performance and outperforms strong baselines by a large margin, especially on datatests with more complicated KB information.

The internet of things (IoT) has been attracting growing attention in recent years. As one potential technology used for IoT sensor network, Bluetooth Low Engery (BLE) is becoming one of the most anticipated solutions to establish IoT networks. Whilst BLE based IoT devices are becoming increasingly popular, security becomes an unavoidable concern not only for the devices by themselves but also for the whole network constructed with BLE devices.

In this thesis, we investigate the security issues on devices and we study the potential security issues for BLE-based IoT sensor networks. In order to explore the security on BLE scatternet, we explore how to form a BLE scatternet, how to implement an appropriate routing algorithm specifically for this network and what kind of potential attack this network may encounter often.

The research endeavor makes four contributions to the security field of intrusion detection on devices and IoT sensor networks. First, we propose a hardware-based intrusion detection approach called CONtrol-flow VERification SystEm (CONVERSE), which ensures control-flow integrity by verifying the destination of control-flow branches at runtime. Many techniques exist for an attacker to alter control-flow to trigger malicious behavior, such as stack and heap overflows which overwrite a return address or function pointer. By verifying branch target addresses at runtime, security exploits can be detected as illegal control-flow. Secondly, we propose an approach for scatternet formation and multi-hop routing for BLE scatternet. We define procedures for device discovery, communication between piconets and forming multi-hop scatternet. Thirdly, we improve the routing algorithm for BLE network in step 2, and present BLE Scatternet Battery- Aware Routing (BSBR), a power aware routing mechanism based on Dynamic Source Routing (DSR) which can be applied to BLE-based Mobile Ad-Hoc Networks (MANETs). Furthermore, we study the impact of battery exhaustion attacks on BLE-based networks. In order to defend against this type of attack, we propose an intrusion detection and prevention approach requiring the suspicious nodes to switch roles with its connected nodes after a pre-determined time. If the suspicious node is identified as a malicious one, it will be blacklisted to prevent future attacks.

Verification of modern digital systems can consume up to 70% of the design cycle. Verification engineers must create formal properties which reflect correct operation from design specifications or other documents. This process of creating formal correctness properties from textual descriptions is a laborious task which is difficult to automate.

In this work I investigate the creation of formal verification properties from textual descriptions written in natural language. I present two approaches that utilize natural language processing (NLP) and machine learning techniques for the automatic generation of verification properties. In the first approach a set of correctness properties expressed in natural language, called natural language assertions (NLAs), is divided into subsets of structurally similar sentences. A generalized formal property template for each subset is used to provide a mapping from each sentence to a well specified verification property. Experimental results show that this methodology reduces the number of formal properties which must be manually created by up to an order of magnitude.

In the second approach I create a custom attributed formal grammar which captures the English semantics of a temporal tree logic. A translation system is implemented which uses this attribute grammar to perform a semantic parsing of NLAs. Attributes for each grammatical production in the parse tree are then used to build a fully specified formal property for each NLA. Experimental results show that valid formal properties are generated from English NLAs in over 90% of test cases.

In evaluating the translation system it was observed that translation rates for NLAs are strongly dependent on the quality of the formal grammar used. High translation rates require a finely tuned custom grammar. To facilitate the creation of such a grammar I propose a new learning algorithm which automatically generates custom attribute grammars from a small set of NLAs and formal properties. This machine generated grammar is used in the NLA translation system to create formal properties from NLAs taken from two design documents for designs in the same product family. Experimental results show that the learned attribute grammar is of sufficient quality to successfully translate up to 88% of NLAs.

Background

Methods

Results

Conclusions