The Erlang-Red Project
Erlang-Red provides a visual low-code environment for creating Erlang architectures inspired by concepts of flow-based programming (FBP).
Erlang-Red incorporates the flow editor designed and built by the Node-RED project. That code is licensed under the Apache-2 license.
Additionally Erlang-Red the project provides a visual flow test-suite which is a collection of Node-RED flows that test the core functionality of inbuilt Node-RED nodes. This allows similar projects as Erlang-Red to ensure compatibility with existing Node-RED functionality.
Erlang-Red is open source software hosted at GitHub.
Background
Flow-based programming (FBP) is a programming paradigm largely accredited to J. Paul Morrison. It is related to Data flow programming. Both paradigms focus on data flows, where computation becomes “black boxed” into independent computational units, visually represented by a rectangle box. Computational units receive data and pass out data, which in turn is passed to other computational units by the underlying architecture of Erlang-Red.
Node-RED is an open source project originally developed at IBM. The core developers continue to work on Node-RED having founder FlowFuse to provide enterprise services centred around Node-RED. Node-Red has had over ten years of active development and is mainly targeted towards (I)IoT and home automation solutions.
Node-RED was designed to support Industrial Internet of Things, with architectures focussing on ingesting many millions of small messages being generated by hundreds/thousands of tiny devices. Message flows are modelled and allow for data packets to be directed to silos for storage or further handling.
Node-RED is an inspired FBP tool and does not implement classical FBP. Classical FBP defines specific requirements such as output messages solely being sent to one output node and that a node can have multiple inputs. Both these requirements are not meant by Node-RED.
Erlang-Red is an attempt to bring the concepts of FBP to Erlang. Others have attempted to apply the ideas of FBP to Erlang however, to the best of my knowledge, no one has attempted to provide a visual environment for flow-based programming in Erlang.
Motivation
Erlang-Red was motivated by an interest in bring the ideas of FBP to Erlang and to bring the ideas of visual FBP to a new developer group. Secondly it is an attempt to create visual-flow-based code that is agnostic to the underlying programming language: visual flows can be executed either using Erlang-Red or Node-RED and are maintained separately from both.
Erlang-Red is explicitly an inspired-by-FBP tool, it does not aim to implement FBP in its classical form. However Erlang-Red comes classic FBP closer by implementing independent processes, on the other hand, Erlang-Red allows outputs from nodes to be passed to multiple other nodes.
Unlike Node-RED, Erlang-Red does not have a specific application instead it aims to provide a generalised tool for implementing data flows. A number of limitations make it unsuitable for specific use-cases but these limitations are related to resource, efficiency or incomplete implementation rather than the underlying concepts of Erlang-Red.
Terminology
Terminology used here.
Flows consist of nodes connected by wires, data flows between nodes along these wires. Flows are static descriptions of data flows that are later implemented in a specific programming language. Dynamic behaviour arises as data actively flows through architectures based on static flows[1].
Nodes are the computational units that process and modify data packets. Nodes have message queues, handling messages in a first in, first out sequence. In Erlang-Red, nodes are modelled as processes and each node is defined by its own Erlang module.
Messages are data packets, normal ranging in size from a few bytes to somewhat less than one megabyte. Message size is limited by memory and the speed of cloning messages. These messages are passed between nodes, being modified at will by nodes. Each node receives its own unique copy of a message.
Data does not constantly flow through a flow, therefore it is useful to think of flows as a set of pipes through which data may flow. The alternative approach is to think of wires as always having a value. This concept of wires is best demonstrated by Blender nodes or NoiseCraft.
Flow editor is the existing Node-RED frontend that being utilised by Erlang-Red. The flow editor is Javascript executing in the browser based on release 4.0.9 of Node-RED. For Erlang-Red, changes have been made to the flow editor. Those changes were needed as not all Node-RED functionality has not yet been implemented.
Erlang-Red: Visual Flow Based Programming for Erlang[2]
Erlang-Red is a first attempt to create visual low-code flow-based coding environment that attempts to be programming language agnostic. Even though Erlang-Red is implemented in Erlang, the learnings created can be applied to other programming languages.
Erlang-Red aims to take advantage of Erlang concepts such as independent lightweight processes, message passing and data immutability to best implement the core concepts of FBP.
Erlang-Red would not be possible without out the many years of development invested into Node-RED. By duplicating the functionality of a set of Node-RED core nodes, many learnings made by the Node-RED team have implicitly flowed into Erlang-Red.
These learnings are the tacit knowledge of those who have helped to shape Node-RED. Erlang-Red benefits from that.
Visual First Development Approach
In deciding to duplicate the functionality of Node-RED, it become clear that I could develop a set of visual unit tests that tested the existing functionality of Node-RED core nodes. These tests could be created visually using Node-RED as unit-test creation tool.
Node-RED does not provide nodes for visually unit testing flows, so I created by own set of unit testing nodes to ensure that functionality was implemented correctly. These nodes became the first nodes to be usable in both Node-RED and Erlang-Red.
These unit tests have been extracted from Erlang-Red and are hosted in a separate repository allowing others to use these test flows to port Node-RED to other programming languages. The intention is to create a standardised collection of test flows to define a visual programming language based on Node-RED[3].
Architecture
Creating the Erlang-Red architecture was guided by the need to replicate the existing functionality of the Node-RED core nodes. However each node had its own requirements on that architecture and therefore the architecture grew organically as more nodes were implemented.
Since message passing and independent processes are builtin, I did not have to simulate those concepts, instead I use the gen_server behaviour and the corresponding handle_cast , handle_call and handle_info functionality.
Early on, as a convention, I made the decision to use handle_cast for inter-node communication, for example, when a node sends another node a message, it calls the handle_cast({incoming, Msg}) function with the process id of the other node.
System messages are transferred via the handle_info function, for example, handle_info({stop}) when the system shuts down all nodes in a flow. Finally when something needs to happen synchronously, the handle_call functionality can be utilised. This many either be system or inter-node communication.
These design decisions have been wrapped into the ered_node behaviour that itself implements the gen_server behaviour. The ered_node behaviour has the responsibility of passing on messages to the modules implementing the nodes. Each node module is required to implement three callbacks: start/2, handle_msg/2 and handle_event/2 with handle_event being invoked for system messages.
The simplest node that best demonstrates these ideas is the junction node which represents a junction point in the flow editor. A junction node passes through unaltered all messages it receives to all the nodes connected to it.
Nodes will never die
Processes are used to represent nodes. I decided that these processes should not die because they have the responsibility of updating the flow editor frontend. For example, with status information on the node, or to update counters for messages received. Instead nodes, if required, should spin up new processes for handling core functionality of the node.
This meant that manager modules were created. Managers being linked to the respective sub-processes which might fail and, if they do, the manager communicates that failure to the original node (that is, process) via a system messages. The node then updates the visuals of the node in the flow editor.
The node can decide what to do, whether to restart the failed processes or whether to die itself. Which is the case for the supervisor node which uses the supervisor behaviour to supervise nodes. Because all nodes use the gen_server behaviour, any node can be supervised by any supervisor node. (Note: two supervisors cannot supervise the one and the same node.)
Supervisor Node
However when a supervisor node is supervised by another supervisor node, it must die to indicate to the supervisor “above it” that its supervisor has died. A supervisor node implements the same “manager” pattern as describe above, therefore it does not die when its subprocess dies. The subprocess for a supervisor node is a “supervisor process” which implements the supervisor behaviour to supervise the processes representing the nodes that the supervisor node is meant to supervise (sic!).
The node not dying then becomes a problem because the supervisor node does not restart the node because it does not know that the node has failed. The simplest solution was failure - the supervisor node is the only node (at time of writing) that dies when its subprocess fails.
On the other hand, if a supervisor node is a “standalone” supervisor, that is, it has no parent supervisors, then it will not die but neither will it restart its supervisor subprocess.
The supervisor process dies when the intensity-period setting is reached by node failures. The supervisor node is made aware of that (via a system message) and changes the status of the node in the flow editor to a status of dead. Additionally the supervisor node sends out a message indicating that it is “dead”.
What has died is the supervisor behaviour sub-process not the node itself, however the supervisor node emulates this behaviour and does not restart the process. Instead, a restart (for a standalone supervisor) can be triggered by sending the supervisor node a message with the action restart.
Thus it is possible to restart processes managed by a supervisor either by using another supervisor node or by using a supervisor flow that is triggered once a supervisor nodes sends a message complaining that it has died, as demonstrated by this flow. (Or even both methods simultaneously, although that is no advised.)
Data Immutability
A further feature of Erlang that is being utilised is data immutability. In FBP messages are independent, two messages are never alike, even when generated by the same node[4]. Each node receives its own unique copy of the message and may modify the message without altering another nodes copy of the same message.
Erlang-Red encodes each message as an map structure and nodes can read from this structure. Each modification of the map structure creates a new map. The BEAM VM ensures that the map structures are duplicated as necessary and modified accordingly.
Thus a further feature of FBP is implemented by the BEAM VM without having to programmatically define that within Erlang-Red. Node-RED codebase provides code helpers to emulate this behave in NodeJS.
Flow editor
The browser component for designing flows is taken from Node-RED directly. The in-browser editor is stable, being based on jQuery making it lightweight (compared to other web frameworks such as React) and portable (jQuery has been around for 18 years and runs on all notable browsers known to humankind).
As there are many hidden features, emulating the functionality of the Node-RED flow editor would take many years. Additionally, the flow editor has been designed and not built: much thought has gone into usability for both experts and novices, and additionally into the appearance of the editor.
I took an approach of developing a set of Cowboy routes to imitate the necessary API calls to make the flow editor function. Not all functionality has been implemented, for example, context and data storage has not. Deploying and restarting a flow, it all works. Code for executing all defined unit test flows has been added, making it possible to visual unit test the entire workings of Erlang-Red.
Erlang, Elixir, BEAM me up Scotty
Since Erlang-Red is a broad in scope and ambitious in features, the support of both Erlang and Elixir communities is necessary. Hence a separate package for collecting Elixir code and linking in Elixir libraries has been created.
It took a few iterations to arrive at this solution of having a separate package for Elixir code, however it now provides a well defined path for adding Elixir code to a project that is centred around Erlang as programming language[5].
JSONata transformational language
In visual programming the hardest characteristic to get right is node granularity: what do and do not nodes do. It is the visual representation of the Unix principle of only do one thing and do it well. Visually the limiting factor is the user interface: just so many drop-downs and fields can be fitted into a browser window (for example).
Node-RED has a function node for executing Javascript code, this can be used to perform minor operations of data structures. For example, retrieving the last element of an array structure, counting the number of members of an array structure, and similar.
However using a function node each time minor operations are necessary soon clutters the flow definition. There are two possibilities: introduce a node for each of the most basic operations or alternatively add a transformational and computational language to Node-RED. Hence JSONata became a big part of Node-RED.
JSONata is a type of XSLT for JSON structures. A message is Node-RED is a key-value pair structure, hash in Javascript, map in Erlang. JSONata is supported as a field type when setting values on the message via the change node.
The change node explicitly acts on the message it receives and changes the message according to rules defined by the user. Using JSONata allows for these changes to be complex and therefore not needing an extra function node.
Since I have the aim of being fully compatible with Node-RED flows, I needed to implement JSONata. This is a further example of Erlang-Red implicitly learning from Node-RED, gaining from the active development of Node-RED.
To implement JSONata in Erlang-Red, I decided to create a language grammar for JSONata (none existed at time of implementation) and created a YECC parser to implement a basic JSONata dialect. The scope of the parser limited to what I require to create test flows for testing functionality. As the set of test flows grow, so will the functionality of the Erlang JSONata parser.
Also I decided that the Erlang JSONata parser would not do the computation of the transformation defined, rather the Erlang JSONata parser transforms JSONata stanzas into Erlang code that is then evaluated in the presence of a message.
Conclusion
Erlang-Red is an exploration of transporting the ideas of flow-based programming (FBP) to the BEAM VM. Explicitly Erlang-Red does not implement the concepts of the classical FBP paradigm and instead is an inspired FBP implementation.
Erlang-Red has shown that the FBP paradigm can be realised in Erlang and by extension, the BEAM VM. Erlang-Red with the help of Node-REDs visual flow editor, provides a visual representation of the concepts of FBP making Erlang-Red an educational tool for presenting the ideas of FBP visually. However Erlangs supervisor behaviour can also be visualised using the supervisor node builtin into Erlang-Red.
Since Erlang-Red translates flow description into an Erlang architecture, Erlang-Red becomes the coordinator between the flow editor and the BEAM VM. Recreating processes when flows are modified, stopping processes when the corresponding nodes are deleted and creating new processes as nodes are added to flows. This makes Erlang-Red a visual tool for creating BEAM process architectures.
Finally by defining of a collection of test flows for testing existing node functionality in Node-RED, Erlang-Red is a project which encourages others to start similar projects for other programming language. Thus creating a body of flows that are programming language agnostic.
Much as the brain is a static collection of neuron and only when a thought occurs does the dynamic nature of the brain come to the fore shine. ↩︎
Node-RED calls itself Low-code programming for event-driven applications. I prefer Visual Flow Based Programming because there is no such thing as “code” or “low code” when thinking in terms of FBP. The ideas behind Erlang-Red are based on data-flow programming and flow-based programming, the focus is on data not code. ↩︎
Imagining a better future is easy, visualising that future is hard. ↩︎
In classic FBP messages are linked to the nodes that generated them implying that messages are “owned” by the nodes that created them. This is not implemented in Erlang-Red. On the other hand, the http-in node is a near exception to that rule: a message generated by an http-in node has properties that make it clear the message was generated by an http-in node. ↩︎
It has been suggested to rename the project BEAM-Red, however there is nothing stopping someone from creating Elixir-Red as an Elixir project. Plus Beam-Red sounds too much like an episode of Star Trek. ↩︎
Comments powered by giscus
The author is available for Node-RED development and consultancy.