Three years Node-RED: A Retrospective
Follow up to my two and one year retros, my third year using and misusing Node-RED.
TL;DR: AI has not wiped us off the face of the Earth and there even seems to be an increased interest in Flow Based Programming for creating Agentic Workflows to combine AI Agents into swarms of intelligence.
This year has been focussed on two main projects for me: my mind map implementation and the start of the how hard can it be? Erlang-Red project. This is then a two-and-half year Node-RED retrospective and a half-a-year Erlang-Red retrospective.
Mind Map - Visions of a Mind
In my first year retrospective, I presented the mind map I created in Node-RED. I wrote that the idea came to me because flows appeared, to me, as the inside of my mind. I began to create specialised mind map nodes and kept working on it. The mind map has now gained many new features, mainly centred around drag & drop content onto the mind map and into my mind.
I can now drop videos, images, audio, text onto my mind map and have nodes created with previews and/or containing text. I mainly drag bookmarks onto the mind map and the application does the work of retrieving the corresponding content and adding it to the mind map.
So if a drag a bookmark that points to an image, my mind map will retrieve the image, store the image, create a node with a link to the local copy of the image and the show me the node in my mind map.
Last year:
This year:
Content is not stored in the flows.json, instead an HTTP-in node proxies the content between disk and browser. Storage is a separate directory within the Node-RED installation and nodes use relative links to access that content.
I added Artificial Intelligence for text classification and summarisation plus additional functionality can convert text to audio or audio to text:
There is a beta version online at beta.openmindmap.org that does not support many features since most features are server backed and servers still cost money. Money that I do not have but perhaps others do and would like to share.
Weirdly enough the backend flow that interfaces with the AI features and proxying the content to the mind map nodes, is it self a flow within my mind map. My mind map is a Node-RED flow using specific mind map modes being executed by Node-RED and not a separate application powered by Node-RED.
A recent bit of content that I added was Hacker News stories which are now added as graph trees. The flow for that is one of the first flows that runs on Node-RED, Browser-Red and Erlang-Red - it is the first universal flow.
Erlang-Red
Erlang Red started with an announcement of Python Red on the Node-RED forum. The idea is to create a python backend for the Node-RED flow editor. I commented on py-red and pointed out that its a great idea but that Erlang is perhaps a better language for representing flow-based programming - mainly since Erlang has in-built message passing, lightweight process that can represent nodes and immutable data structures for ensuring message ownership.
It made me think and basically I thought how hard could it be? After all, Erlang has a cowboy as web framework and the APIs of the flow editor are quickly created. From experience of putting together the Browser-Red editor, I knew that the APIs required by the flow editor were not that complicated nor numerous.
Project Structure
First thing I did was to ask over at the Erlang Forum whether anyone else had attempted something similar. There had been an attempt to bring FBP to Erlang via a textual solution no visual attempts. There was much good advise on how to structure the Erlang codebase and thankfully I followed that advise.
As the project grew, I started to push things out into separate libraries so that the Erlang-Red codebase concentrates on implementing the FBP concepts/ideas, while support functionality for Node-RED behaviour is out-sourced into Erlang libraries:
- JSONata support,
- Javascript-like object accessing and
- Elixir support.
So far, JSONata was the hardest nut to crack in creating Erlang-Red. I have to support it if I want to be 100% compatible with Node-RED but how do support something that is intrinsically rooted in the Javascript world?[1]
I have a naming convention that everything related to the project has a ered_ prefix. That allows Erlang-Red to be mixed into other existing Erlang projects.
100% compatible with Node-RED
From the beginning I wanted to be 100% compatible with Node-RED functionality. Many folks view this as impossible and indeed, with 5555+ Javascript node packages, it is impossible. I will not be able to support every flow using every node defined in Node-RED but that is not the point.
The point of “100%” is the core node functionality. All the nodes that come with Node-RED should be supported and function exactly as they do in Node-RED. This has been a challenge but there are many advantageous of taking that approach.
First: relax, stop thinking - I don’t need to decide what node functionality should be implemented — each core node has its well-defined functionality and all I need to do is copy that. That is great because I hate making decisions where the outcomes are unclear.
Second: ten years of development - Erlang-Red benefits from the ten years of development put into Node-RED. I have come to understand the design decisions made in Node-RED and am happy to port those to Erlang-Red. Everything from UI design to node functionality, it’s all usable in Erlang-Red. To recreate the flow editor would have taken years, instead I embrace it and have the benefits of the same UIs as Node-RED.
Third: testability - I soon realised that if I create test flows in Node-RED that test specific functionality of core nodes, then if I implement the functionality in Erlang-Red, those test must succeed in Erlang-Red. This made development somewhat like programming by numbers!
Flow Test Suite
To support test flows, I created the first node package that works both with Node-RED and Erlang-Red. The unit-test package provides the basic assert functionality to ensure that messages are sent, that message contain specific values and that debug and status notifications are generated.
Using those nodes, I first create clear visual unit tests[2] in Node-RED, then implement the corresponding functionality in Erlang-Red and finally ensure the tests succeed in Erlang-Red. I then realised that perhaps these tests are also useful for the Node-RED team for testing new releases and doing regression testing.
The test suite is now a standalone repository of more than 250 tests. I integrate the collection using FlowHub.org which is my own creation for managing flows across installations of Node-RED. FlowHub.org is a visual interface to GitHub which is fully integrated into Node-RED and Erlang-Red.
FlowHub.org is something that I started creating two years ago because I soon realised that Node-RED does not have a visual comparison tool[3]. As a developer, I am used to having tooling for code versioning and code comparison. These are provided by Node-RED however in a textual form, which for me, makes no sense if I am to programming visually. That is not a criticism, this became an itch for me.
I have always considered Node-RED as an IDE for coding visual programs and with Erlang-Red, I have come to clear realisation that target audiences differ. Node-RED has two sets of target audiences, I intend for Erlang-Red to have a single audience.
Target Audience(s)
For me, Node-REDs two target audiences are the non-coders that use Node-RED mostly for home automation and the developers that create node packages to extend the functionality of Node-RED. There is a certain overlap in these two groups but the non-coders are definitely in the majority.
It took me a long time to realise this. Why is this important for me? Because I always saw Node-RED as a coding tool, something akin to an IDE for putting together code. I therefore thought that the folks using Node-RED were also coders. Hence I created a serious of tools for making coding in Node-RED simpler - at least for me:
- NodeDev package for creating nodes as flows within Node-RED - no more switching between Emacs and Node-RED when developing nodes for Node-RED,
- Introspection to gain insights of the structure of flows within the flow editor. Thus better understanding message flows. The package now includes message tracing to transparently monitor server message flows,
- Flow Compare for visualising changes made in the flow editor compared to what is deployed on the server,
- Flow 2 UML for automatically generating Mermaid UML - flow charts - diagrams from Node-RED flows.
All this tooling might be useful for developers but definitely not for casual home automation users of Node-RED. I never realised this until this year.
In creating Erlang-Red, I made the conscious decision to build a tool for developers. My aim is to create awareness within the Erlang community of a visual flow based programming paradigm that is well matched to Erlang and that can be useful to create Erlang code that everyone[4] can understand.
Erlang Behaviours
Erlang has many features that make it different to NodeJS and generally other programming languages. Erlang has different terminology that needs to be learnt, for example “behaviours”.
Erlang behaviours represent design patterns that are often required in Erlang applications. They have been distilled from the various and many applications of Erlang within the telecommunications world and have been found to be useful. Ignore them at your own risk!
One for example for a behaviour is the supervisor.
As independent lightweight processes are a key feature of Erlang, managing and monitoring of those processes — processes that comprise the application — is a mainstay of any Erlang application. The supervisor behaviour is a behaviour that makes this simpler, by allowing programmers to define their processes and having the supervisor manage the lifecycle of those processes: start, stop and restart processes as necessary.
But — the supervisor behaviour presented me with a dilemma.
Do I hide the behaviour away since Node-RED/NodeJS does not have anything equivalent. But then how do I manage the processes for nodes? Do I make a random decision on when and how nodes are restarted and hide that from the user?
On the other hand I could make the behaviour a first class citizen in Erlang-Red and create a visual element for it. But this would make Erlang-Red flows incompatible with Node-RED and thus break my aim of 100% compatible — compatibility goes both ways after all.
I decided to allow users to create their Erlang architectures in Erlang-Red and have a supervisor node that could be configured as any other node. I created a separate node package which can be installed in Node-RED.
Hence my target audience became Erlang developers since an understanding of the supervisor paradigm is now required to fully use and understand Erlang-Red.
Cross-pollination of Ideas
This is an example of the cross-pollination of ideas. Should the supervisor node work with Node-RED nodes? Do Node-RED nodes fail? They do but they are not processes within Node-RED. When a node fails, the entire flow fails. Or rather the message being passed through the flow fails. Rarely if ever does Node-RED fail completely. As each message arrives, Node-RED just performances the steps defined in the flow.
In that sense, a flow in Node-RED is an algorithmic plan for what steps are to be performed on a message. If something fails, the message is tossed out, i.e., sent to the debug panel and the next message is handled.
Erlang-Red defines a collection of processes that represent the flow and each process represents a node in that flow. As messages arrives, they are passed amongst the processes eventually reaching an end node and then being dropped. If a process dies as the message are being passed around, it prevents messages from being passed through the flow. So it is or it isn’t important that processes recover. But who decides that? Erlang-Red or the user who designed the flow?
For me the answer is the user. The user is given the responsibility because my target audience are programmers who aim to design Erlang self-healing architectures.
After the supervisor node, I created the event machine node, the state machine node and the generic server node, each representing a behaviour defined in Erlang. I ensured that all these nodes work with the supervisor node, so that if they fail, they are revived by the supervisor node.
Even supervisors of supervisors of supervisors of … is possible to be designed in Erlang-Red.
Breadboard Programming
Beware this is mind dump of work in progress.
Prototyping but done differently.
Working on Erlang-Red and creating flows that were testing features of Erlang-Red made me feel somewhat as if I was breadboarding some electrical components together to create a Frankensteinian creature called Erlang-Red.
An example breadboard flow:
Shown in the breadboard is the creation of a TCP server that only allows one user to connect. The implementation uses the state machine behaviour and shown are three attempts at getting it right.
This for me is the point: I can prototype my code without have on alter it, I can keep all prototype solution around and can build on each or throw them out. That is because copying & pasting code is semantically cheap: code is heavily encapsulated, copying code has less risks.
Nodes are independent of one another thus multiple copies of themselves are not interrelated. Nodes are affected by and affect, messages passing through them. Thus it is possible to keep old flow around. In the three variations shown, the only thing that needed changing was the tcp port used for testing. I copied the entire group of nodes that represent the interaction with TCP node combined with a state machine and all I had to change with the port number that the TCP-in node was listening to. I could then alter the flow to have different behaviour.
In my opinion, that makes breadboard programming different to prototyping where code is developed or completely replaced but rarely are multiple copies of the same prototype maintained concurrently.
Some more background
I did an interview on Erlang-Red over at TADSummit. There I also talk about the Node-RED inspiration and my aims of promoting Flow Based Programming to another community. Also the original article on Erlang-Red provides some technical details on the Erlang architecture.
Concluding
Thankfully I finally made to the bottom of this page. There is much I could have added, hopefully nothing that I should have removed.
There is still much to be done for visual flow based programming to make any major impact but that does not mean one should not continue to explore possibilities.
This year, for me, has been filled with many points of wanting to give up and move onto something else. But thankfully I did not find anything else in my mind map other than my mind map.
I look forward to the day that my mind map runs on Erlang.
Spoiler: Erlang parser definitions are wonderful. I ended up creating a parser that converts JSONata stanzas to Erlang code equivalences. ↩︎
These visual unit tests can also be created by anyone, so that “non coders” could create tests that ensured that Node-RED keeps working. ↩︎
FlowHub.org is also completely 100% coded in Node-RED, so it is itself a visual flow based program. ↩︎
“Everyone” here includes project managers, product owners, quality assurance, generally anyone involved in an Erlang project. ↩︎
Comments powered by giscus
The author is available for Node-RED development and consultancy.