• Corpus ID: 239015891

Lifting DecPOMDPs for Nanoscale Systems - A Work in Progress

@article{Braun2021LiftingDF,
  title={Lifting DecPOMDPs for Nanoscale Systems - A Work in Progress},
  author={Tanya Braun and Stefan Fischer and Florian-Lennert Adrian Lau and Ralf Moller},
  journal={ArXiv},
  year={2021},
  volume={abs/2110.09152}
}

• Tanya Braun, Stefan Fischer, +1 author Ralf Moller
• Published 18 October 2021
• Computer Science
• ArXiv

DNA-based nanonetworks have a wide range of promising use cases, especially in the field of medicine. With a large set of agents, a partially observable stochastic environment, and noisy observations, such nanoscale systems can be modelled as a decentralised, partially observable, Markov decision process (DecPOMDP). As the agent set is a dominating factor, this paper presents (i) lifted DecPOMDPs, partitioning the agent set into sets of indistinguishable agents, reducing the worstcase space required, and (ii) a nanoscale medical system as an application. Particularly in times of medical crisis, precise and efficient diagnostic tools are invaluable. The recent development of DNA-based nanonetworks shows promising results as a fast and robust diagnostics tool (Lau, Wendt, and Fischer 2021) and a treatment method for diseases using nanobots in the human body (Lau et al. 2019). Such tasks are realised by a swarm of thousands of nanodevices collaborating to compute a diagnosis or treat a disease. However, a network’s environment is extremely heterogenous. Thus, the nanodevices have only local information about the system, which they are not able to consolidate into a global state, with their communication and computation capabilities highly limited and local information subject to noise. Under these conditions, a designer of a nanonetwork has to set up nanodevices functioning according to a certain plan or policy. In addition, the designer needs to assess, e.g., the network’s robustness regarding a diagnosis or the success rate of a joint action of medicine delivery. Therefore, this paper formalises the setting as an offline decision making problem, specifically, a decentralised partially observable Markov decision process (DecPOMDP). The formalisation fits well as a network contains a set of collaborating nanodevices, i.e.,agents, with limited capabilities in a partially observable environment whose transition can be approximated with a stochastic process. Computation time in DecPOMDPs depends exponentially on the number of agents in the worst case, which is problematic as nanonetworks can have hundreds of thousands of agents. Copyright © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. However, only a limited number of types of nanodevices exists, thus, partitioning the set of agents into subsets of agents whose behaviour is indistinguishable. The notion of lifting refers to the idea of efficiently handling sets of indistinguishable objects using representatives (Poole 2003), reducing the theoretical dependency from exponential to polynomial w.r.t. these set sizes. Here, we apply lifting to the agents of a DecPOMDP. Specifically, the contributions of this paper are twofold: (i) lifted DecPOMDPs with a theoretical analysis in terms of space requirements and (ii) a nanoscale medical system as an application, forming a first step towards our long-term goal of a full formalisation combined with a lifted solution approach that allows for quantifying a joint action’s success rate. Lifting has been successfully used for probabilistic inference (see, e.g., Van den Broeck et al. 2011; Ahmadi et al. 2013; Braun and Möller 2018; Holtzen, Millstein, and Van den Broeck 2019) as well as online decision making (Nath and Domingos 2009; Apsel and Brafman 2011; Gehrke et al. 2019; Gehrke, Braun, and Möller 2019), which uses decision and utility nodes in a lifted probabilistic graphical model (PGM). Lifting can even bring tractability in terms of the set sizes of indistinguishable objects (Niepert and Van den Broeck 2014). In offline decision making, lifting has been used in calculations for relational descriptions of (PO)MDPs: Firstorder MDPs (FOMDPs, Boutilier, Reiter, and Price 2001; Sanner and Boutilier 2009) have a representation based on the situation calculus (McCarthy 1963). Sanner and Kersting (2010) use lifting for pruning indistinguishable policies in their solution approach for a partially observable FOMDP. Factorised FOMDP assume a factorisation of its representation, blurring the line towards online decision making with the factored representation (Sanner and Boutilier 2007). First-order open-universe POMDPs follow the open-universe assumption for the first-order description of an environment using Bayesian logic as a basis (Srivastava et al. 2014). Another first-order representation is the language of independent choice logic that also allows for a set of agents (Poole 1997). To the best of our knowledge, we are the first to consider lifting the agent set, which finds its application in nanoscale systems where large groups of indistinguishable nanodevices act jointly in an environment. We leave the environment encoded by a non-lifted joint distribution in this paper since the application at hand does not lend itself to a highly structured model. Nonetheless, dealing with structure in the environment, relational or factorised, presents an exciting avenue for future work to extend its applicability. The remainder of this paper is structured as follows: We start with an introduction to nanoscale medical systems and recap offline decision-making from MDPs to DecPOMDPs. Then, we present lifted DecPOMDPs, followed by a discussion of a lifted DecPOMDP modelling a nanosystem and next steps towards solving such a problem. We end with a conclusion. Nanoscale Medical Systems DNA-based nanonetworks have been proposed as an alternative to polymerase chain reaction, PCR for short, for detecting arbitrary diseases on the basis of DNA. In this scenario, a disease sample is mixed with a medical nanosystem that computes a programmed function depending on environmental parameters to decide if a disease is present. This section explains the basic mechanisms and ideas behind this novel technology and sets up how DecPOMDPs can be used as a model for it. In general, a computational process at nanoscale is subject to resource constraints, and collaboration between nanodevices might be necessary to achieve a goal (Akyildiz, Brunetti, and Blázquez 2008). In 1982, Seeman first proposed DNA as a construction material for nanoscale systems. Based on this idea, Rothemund (2006) developed the DNA-origami method, which allows for creating almost arbitrary shapes using DNA at nanoscale. In 2009, Andersen et al. created a box with a controllable lid using the DNA-origami method. These boxes serve as a basis for medical nanosystem technology. They can be filled with either medication or DNA-tiles that serve as an input for a computation (Winfree et al. 1998). Certain DNA-tile systems form a Turing-complete computational model and are much more capable than most widely used medical diagnostic tools (Lau et al. 2019). For a detailed introduction, we refer to Winfree et al. (1998). Figures 1a and 1b show an example system of DNAtiles that computes a 4-bit and operation. The blocks in Fig. 1a represent DNA-tiles. They are modelled as non-rotatable blocks with colour-coded glues at possibly all sides. The number of black boxes represents the strength of glue and a label next to it a condition. Tiles with the same number of boxes and identical labels can form a binding. That binding is subject to an environmental parameter called temperature τ . The temperature τ encodes the necessary number of fitting glues over all neighbours to form a stable binding. In the presented example, the temperature is 2, and all tiles need at least two neighbours to stably bind together. An assembly process begins with a seed-tile σ, shown in Fig. 1b at the right. Due to the temperature requirement of 2, three tiles can bind to the seed-tile σ: tile T, σ d