Okay okay, what do I mean by a programming language for RNNs? RNNs are usually trained on data using a cost function. In contrast, computers are programmed to run a sequence of operations in an algorithm that manipulates data in a precise way https://t.co/4XMeCckQ9T. pic.twitter.com/XHn8wkydp0
— Jason Kim (@jason_z_kim) March 11, 2022
Check out the whole thread.
Here's the abstract for the linked article:
From logical reasoning to mental simulation, biological and artificial neural
systems possess an incredible capacity for computation. Such neural computers
offer a fundamentally novel computing paradigm by representing data
continuously and processing information in a natively parallel and distributed
manner. To harness this computation, prior work has developed extensive
training techniques to understand existing neural networks. However, the lack
of a concrete and low-level programming language for neural networks precludes
us from taking full advantage of a neural computing framework. Here, we provide
such a programming language using reservoir computing -- a simple recurrent
neural network -- and close the gap between how we conceptualize and implement
neural computers and silicon computers. By decomposing the reservoir's internal
representation and dynamics into a symbolic basis of its inputs, we define a
low-level neural machine code that we use to program the reservoir to solve
complex equations and store chaotic dynamical systems as random access memory
(dRAM). Using this representation, we provide a fully distributed neural
implementation of software virtualization and logical circuits, and even
program a playable game of pong inside of a reservoir computer. Taken together,
we define a concrete, practical, and fully generalizable implementation of
neural computation.
