ijcnn05 montreal workshop, 19:00-22:00 thursday 04aug05 nns, bio- and neuro- informatics junk dna...

IJCNN05 Montreal Workshop, 19:00-22:00 Thursday 04Aug05

NNs, Bio- and Neuro- Informatics

Junk DNA and Neural Networks:conjecture on directions and implications

Mr. Bill Howell [email protected](use Menu selection “View->Notes” to see the notes which accompany many of the

slides in this MS Powerpoint presentation)

The views expressed in this presentation are personal and speculative. They are in no way related to the research, policies, viewpoints, and programs of my current employer, the federal government department “Natural Resources Canada”.

To the best of my knowledge, there is no work underway or planned in this area within the federal government at this time.

Bill Howell

International Joint Conference on Neural Networks 2005, Montreal NNs and Bio-/Neuro- Informatics

Outline

1.Revolution beyond the "central dogma of biology" – DNA: function beyond gene-protein through “junk”– Junk = non-protein coding, incl non-exon, epigenetics, other?

2. Related trends with NN architectures and processes• Structure, function, messaging, dynamic transitions• Learning, control, planning, behaviour, goals

3. Computational Neuro-Genetic Modelling (CNGM)1. Bio- and Neuro- informatics relevance2. Expanded approach to ANNs

4. Implications for the brain and the mind● Inheritance of vast knowledge, grammar (not just linguistic!)1. Multiple parallel “behaviours & personalities” (computing)2. Evolutionary theory3. Mindcode (highly speculative)

1. Revolution beyond the "central dogma of biology"

2. Related NN trends with architectures and processes

3. Computational Neuro-Genetic Modelling

4. Implications for the brain and the mind

“The gold standard for NNs, far off in the distance, IS the human brain...”


Junk DNA as code

● A revolution in the "central dogma of biology"?– John Mattick (UQueensland), others over several years

● Eukaryotic DNA coding >>> genes for proteins • ~1.5% of human DNA codes for proteins, but most DNA

transcribed to RNA!● poor relation between organism's complexity & # of protein-

coding genes, more consistent relation with non-coding DNA?● DNA expression


Junk DNA as code (cont'd)

– Genes: conventionally thought as literally "assembly language programming" for proteins – perhaps the simplest and lowest level of programming on the DNA?

– Junk -> RNA and micro-RNA, regulatory role● Chromatin – mask/reveals DNA code● Architecture – need precise plans, “drawings” - highly

specific

– Cambrian period bio-complexity explosion (~1 Gy ago)


John Mattick: “Cambrian complexity explosion”


Possible Examples?● Timing circuits in cells/ organism (special emphasis on time,

sequencing, coordination of parallel and sequential processes)– multitemporal, multifractals

• Cell-line specific genes/ structures/ functions – cell signaling mRNA -> [datamining, eg constellations]

• High cross-over rate genes (individual variabilities such as appearances)

• Linguistics – Chomsky, Pinker (1994), note that the distinction between genes and "non-gene" DNA is not emphasized by many authors

● Feedforward - basis of current descriptions of npcRNA

● Feedback from neuron to DNA– A given for regulatory control, but is this limited to

“normal” physiological sensing/ control pathways?– Is there a means for the neurons to direct SPECIFIC

DNA expression (protein or non-protein coding DNA), in a way that isn’t simply physiology, but is directly for specific information processing/ data tasks?

– Could lead to Recurrent Neural Networks (RNNs) that use DNA coding/ “programs” (programming metaphor)- with very powerful advantages!


Neuron to DNA signal & control?


Advantages of Junk DNA as Code● Mattick:

– away from the combinatorial soup of proteins for regulation, towards a highly specific "program" to direct spliceosomes (low side effects, fast, efficient)

– specification of architecture/ growth allows vastly greater complexity of organisms (just what we are looking for with the brain!)

– alternative splicing & "overloading" of genes – assemble protein-coding-RNA in one of several ways, code has different functionality in different cell types

● Other possible advantages - metaphor of computing– Vastly parallel (as fits NNs) – Not just “static” code – dynamic interactions between coding– ??Co-resident junk & protein code (data/ methods -> super-

objects), “calls” to ensembles of NNs


Challenges – extrapolating to the brain● Genes – "relatively easy" to see, identify coding

it's still a work in progress, start/ stop sequences, intron removal etc, how to decide when two conformations are possible?

● Cell physiology -> "event" amoung many others happening in parallel, still can "see" amongst all of the mRNA & cell signalling happening at the time

● Ontogeny (growth) - visibility like cell physiology, but how easy is it to quantify subtle structural changes?

● But what about more abstract processes, and thought? How can one identify these?how to link code & its effect?


What toolsets do geneticists have & need?

● Already a great base & fallback position – current genomics/ bioinformatics/ neuroinformatics

● Need - greater exactness of:– [Reading, precisely changing] DNA code, RNA & epigenetics– Measuring structural changes in neurons & NN ensembles– Also, what are “feedback” mechanisms from NN changes to

DNA?

● Signaling – how good is this for the purposes in mind?

● Computational Intelligence tools? (third section)

2. Related trends with NN architectures and processes



1. Revolution beyond the “central dogma of biology"


NN Architecture & Multi-phase modules:"Crystalline to Gaseous"

● Fixed weight neural networks (IJCNN05 Guang-bin Huang extreme learning machines)

● Echo state networks (IJCNN05 Special Session Rao & Principe)

● Some aspects of:– Neural Gas Models (other name of similar NNs?)

– Chaotic NNs (IJCNN05 SS - Kozma)


Functional overloading

Functional overloading – similar to neuro-modulators and gene networks

● Same NN module is simultaneously part of many different “models” and model trainings

● Variable object inputs/ problem types


Networks, hierarchies of NNs ● Networks, hierarchies of NN ensembles (Mexican?,

ART, SOMs)

● Adaptive Critics and ObjectNets (Werbos, Venayagamoorthy) and agent-like systems

● What is happening to Neurosolutions software? Are there an emergent systems for NN ensembles? (Principe)

● Neurology - Purkinje cells, climbing fibres etc etc


Dynamic transitions● Dynamic transitions - during learning and action

● Searching problem/ solution statespace– evolutionary theory, particle swarms– chaos - ?directed?, stochastic

● Variable object inputs/ problem types - as with functional overloading

● Dynamically reconfiguring structures of NN modules


Learning &Training implications

Previous issues tie in well to Junk DNA/CNGM● manner of selecting pre-specified, powerful starting points for

brain/ systems of ANNs (eg Huang and ?de Jaeger?)● capability of building robust, effective systems of ANNs, and

hybrid systems involving ANNs● don't need a small set of tools, not restricted to a single

learning theory/ method● looking for orders of magnitude faster training plus much more

accurate/ more robust solutions where that is possible, with powerful general learning techniques for unusual/ difficult problems

● for some problems, perhaps we cannot expect fast solutions – evolution of capabilities over much time or many generations may be required

● Meta-learning

– Kenji Doja (2002) – learning parameter initialization/ adjustment


Learning &Training (cont'd)


"Small-world” universal function approximation

● Small world domain – what is the smallest set of NNs, of various functional capabilities (general to specific), that is sufficient to solve most of the problems in a domain of interest?

● How does one restructure/ combine/ train these NN modules to obtain ultra-fast, accurate learning/ control? (eg for control – ObjectNet adaptive critics)

● What happens for a much broader or universal domain? (combine both general & special NN)

● Building from level of abstraction to the next● Converse issue – false confidence in good fits (eg Global

Circulation Models for climate, Valdes @ IJCNN05)


Expanded approach to NNs

● Nik Kasabov, Benuskova, Wysoski (Aukland UTech) - architectures for gene networks, extending the concept to advance NN architectures

● What are the desired capabilities/ opportunities?– Co-resident [code, genes] - beyond OOPS?, new

RNNs?– Data [delivery mechanism] - code segments identify

data (DNA or RNA code keys physically bring data and destination together!), data can drive code & architecture…

– Dynamic structures – switch/evolve in real time! Approximate Dynamic Programming (ADP)

● Hypothesis of bidirectional action:– Junk DNA -> drives neuron states– Neuron states -> initiate junk DNA sequences

● Different object inputs to, and functional behaviour of, a single NN module or architecture of NN modules

● Rapid reconfiguration of ensembles of NN modules along “high likelihood” arrangements, longer term more exhaustive evolution


Expanded NNs (cont'd)


What differentiates CNGMs from current NNs?

● Will they simply be faster/ more accurate?– Like fixed weight NN - much faster system

identification/ learning

● Or are there new capabilities that will arise?– Greater “aptitude” for symbolism?

● Semantics and Logic as emergent properties for very complex systems (abstraction -> soft logic, giving explainability)

– Greater functional/ mapping specificity– Ease of combining networks of NN modules?

● What else? (my feeling: HUGE conceptual advances)


Implications - for the brain and the mind

● Inheritance of vast knowledge (content) - form very specific to general and to highly abstract– Data, procedures, “operating systems”, behaviours …– Selection of the right blends of different levels of

abstraction for inheritance– Eg language - could “know” all words, but it is the

dynamic bindings that give language its power– Far beyond “Nature versus Nurture” (which was

somewhat a ?dysfunctional? discussion anyways - eg ontogeny effects on identical twins, compression effects in relating synapses to inherited DNA). The problem with dichotomies…


Implications (cont’d)● Inheritance of a vast grammar

– As per Chomsky, but not just linguistics - much more fundamental than that, at the connectionist levels as well as at the symbolic/ linguistic levels

● Multiple parallel behaviours & personalities– But will this somehow affect goal identification/ prioritization?

● Problem decomposition/ modularisation, reconstruction– Multiple models that are dissimilar, yet collaborate effectively

and simultaneously on a variety of models of the problem(s) being addressed


Evolutionary Theory● Lamarckian heredity - pass on capabilities (strong arms,

knowledge) developed over course of lifetime

– Easily done for individuals, organisations, society, computers -> learning, charters & constitution, laws etc etc

● Would a Lamarckian evolutionary approach “re-discover” basic principles and laws that have long been established in other areas?– learning (pass on through teaching, apprenticeship,

experience, management fashions)– Economics, organization structure and processes?? etc


Mindcode● "Given that computer code is used to program

computers, then mindcode..."– The perspective here isn't to "program" a child/adult brain by

some external means, but rather to seek an understanding of junk DNA coding (and perhaps other sources of coding such as epigenetics) that may define the basis of our brains from conception. What might such code tell us about ourselves and our history that is different from current psychology, sociology, anthropology, management theory, economics?

● This is pure speculation and fantasy, but I think that it is a useful fantasy to drive lines of investigation, to maintain an awareness of the types of results that we should be looking for (its easier to find something when you are looking for it).


References1. Junk DNA, epigenetics, neuromodulation

Ast, Gil “The alternative genome” Scientific American, vol 292 iss 4, Apr05, pp58-65

· Doja, Kenji; Dayan, Peter; Hasselmo, Michael (guest editors) "2002 Special Issue: Computational Models of Neuromodulation" Neural Networks, Vol 15 Nos 4-6, June-July 2002: Kenji Doya "Metalearning and neuromodulation" pp495-506

· Eddy, Sean R. “Non-coding RNA genes and the modern RNA world” Nature Reviews Genetics, vol 2, pp 919-929, December 2001(from Gibbs)

Gibbs, W. Wayte "The unseen genome: Gems among the junk" Scientific American, vol 289 no 5, Nov03 pp46-53

Ingoglia, N. organizer (American Society of Neurochemistry (ASN)) “RNA Interference (small RNAs): Applications to neural systems” Pre-meeting workshop, ASN Annual Meeting, 14-18Aug04, New York city

· Krichevsky, Anna (Harvard Medical School) - research focussed on neurons: Kim, J; Krichevsky A; Grad, Y: Hayes, G.D; Kosik, K.s; Church, G.M; Ruvkun, G “Identification of many microRNAs that copurify with polyribosomes in mamalian neurons” PNAS vol 101 no 1, 06Jan04, pp360-365

Mattick, John S. “Challenging the dogma: the hidden layer of non-protein-coding RNAs in complex organisms” BioEssays, vol 25 pp930-939, Oct03

· Mattick, John S. (UQueensland) "The hidden genetic program of complex organisms" Scientific American, Oct04 pp60-67. See http://imbuq.edu.au/groups/mattick

2. Related trends with NN architectures and processesa) Guang-Bin Huang, Qin-Yu Zhiu, Chee-Kheong Siew, Nanyang TechU, Singapore

"Extreme learning machine: A new learning scheme of feedforward neural networks" IJCNN04 Budapest

b) Ganesh Kumar Venayagamoorthy (UMissouri-Rolla) "Dynamic optimization of a multimachine power system with a FACTS device using identification and control ObjectNets" (IAS2004 Yikes – I just have the paper and forget which conference!) 0-7803-8487-3/04/$20.00 © 2004 IEEE

c) Roberto.A. Santiago, NW Computational Intelligence Lab, Portland StateU, Oregon "Context discerning multifunction networks: reformulating fixed weight neural networks" IJCNN04 Budapest

d) ?"Co-Evolutionary Learning of Liquid Architectures" Igal Raichelgauz, Karina Odinaev Yehoshua Y. Zeevi, Israel unpublished yet?


References


References

3. Computational Neuro-Genetic Modellinga) N. Kasabov, L. Benuskova, S.G. Wysoski (Knowledge Engng & Discovery Research Institute

(Kedri), Aukland UofTech, New Zealand) "Computational neurogenetic modelling: Gene networks within neural networks" IJCNN04 Budapest

b) Rui Xu and Donald C. Wunsch v- see IJCNN05 references


IJCNN05 References IJCNN05 = International Joint Conference on Neural Networks 2005, Montreal, International

Neural Netwsork Society and IEEE Computational Intelligence Society. Because of the ease of accessing conference papers, many are listed below. This is certainly NOT exhaustive!!

a) Relevant sessions include:– S2 "Computational Neuro-Genetic Modelling" chair Nik Kasabov– S8 "Constructive/Hierarchical Self-Organizing Maps" chairs Ernesto Cuadros-Vargas and

Roseli Francelin Romero– Sa "Computational Dynamical Modeling with Echo State Networks" chairs Yadunandana Rao

and Jose Principe– Sf "Evolvable and Emergent Neural Systems" chairs Seong Kong and Jacek Zurada– P1-Gf "Neural network architectures and structures" chairs: IJCNN05 program chairs

b) Related papers include:1. 1728 "Gene Regulatory Networks Inference with Recurrent Neural Network Models" Rui Xu

and Donald C. Wunsch II, ACIL, University of Missouri-Rolla, United States2. 1016 "Functional Grouping of Genes Using Spectral Clustering and Gene Ontology" Nora

Speer, Holger Froehlich, Christian Spieth and Andreas Zell, Centre for Bioinformatics Tuebingen (ZBIT), Germany

3. 1603 "Modeling Cortico-Subcortical Interactions During Planning, Learning, and Voluntary Control of Actions" Daniel Bullock, Boston University, United States

4. 1257 "Protein Sequence Classification Using Extreme Learning Machine" Dianhui Wang, La Trobe University, Australia; and Guang-Bin Huang, Nanyang TechUniversity, Singapore

ijcnn05 montreal workshop, 19:00-22:00 thursday 04aug05 nns, bio- and neuro- informatics junk dna...

Documents