Pipes, Wires and Nodes: Rethinking Node-REDs terminology
Recently I came across something that made me rethink the Node-RED terminology as I understood it. The insight helped me to better understand Node-RED flows and visual programming tools in general.
It arose from a discussion of alternatives to Node-RED or similar visual programming tools. The initial application was Noisecraft that emulates a digital modular synthesiser, it has no overlap with Node-RED other than its a visual programming tool. The website states:
NoiseCraft is an open source, visual programming language and platform for sound synthesis and music making, with the goal of creating a community for the open exchange of musical ideas. - Source
A typical Noisecraft flow:
A sample of a Noisecraft composition - source.
It reminded me of Blenders node programming interface, which many applications have emulated:
Blender nodes in full action - source
For me those interfaces are characterised by having multiple colour-coded typed inputs, with the colours provide a visual cue for which input may be connected to which output, to aid construction of complex flows. For me this has always been visually complex and I had a difficult time understanding how those “flows” worked - what happens if two of three inputs were solely available? What do the individual inputs even mean? What does the function behind the node do with those inputs?
When I discovered Node-RED, I immediately understood: one or zero inputs and no typing of outputs nor inputs. A single message flowed through a well defined pathway through a series of nodes which represented a computational unit. Why weren’t all flows constructed in this manner?
Node-RED flow: LED display controlled by an I2C bus interface - source
As a programmer that made complete sense but why does the node programming of Blender make no sense? Why does the Blender flow overwhelm me but not the Node-RED? I had never given it much thought until a discussion ensued because of my preference for the grey one-or-zero concept of Node-RED[1]. Why does Node-RED do it differently? Turns out, Node-RED is event driven:
The light-weight runtime is built on Node.js, taking full advantage of its event-driven, non-blocking model. - Source
Which implies that when events happen, data flows otherwise nothing happens. Events are at the core of Node-RED flows. Node-RED flows describe the action handling that should be performed when an event happens. Each event has data attached which flows through a specific pathway defined by the flow.
Herein lies the difference to applications such as Noisecraft: both Blender and Noisecraft are not event driven, they are state machines synchronised to a clock. In the case of Noisecraft clock nodes provide the tick and in the case of Blender its the frame counter that provides the tick (for example).
With each tick, the state machine updates its state by the rules as defined in the “flow” diagram. Is it a flow diagram or a visual program for updating the state of the state machine? One could call it a flow-state diagram since it shows the change state as time flows.
Change
Another difference is what actually changes?
In Node-RED all changes are applied to the data object flowing through the computational units, i.e., the nodes. The flow does not change, data changes. Compare this to Noisecraft and one notices that the flow changes over time, there is no data that changes, it is the flow itself that changes with the tick of the clock. Stop the clock and changes stops.
Node-RED has no clock, instead if there are no events, then change also stops. Events are not consist and predictable, a clock is very predictable and provides stability in an event-oriented world.
Connector Terminology
Most visual programming applications speak of “wires” when they speak of the lines that act as connectors between the nodes. But is that the correct terminology?
What are wires?
For me wires are electrical wires, turn on the switch and there is a value. The value remains until the switch is flipped off and that gives the wire a different but permanent value. A wire always has a value, be it off or on. A wire has currents, ampere and resistance to define its properties. These properties aren’t reflected in most node programming environments.
What are pipes?
I can open a pipe but nothing flows or will flow for a short time but then stop. I can shut a pipe and build up pressure that will need realising at stage. Pipes have a limited space, only so much liquid can flow through them, more will simply not fit or cause too much pressure causing the pipe to burst.
It becomes clear that pipes describe the connections created in Node-RED far better than wires.
A second understanding of pipes that I have is “Unix Pipes” which are connectors of programs to form chains. Each program uses the output of the previous program as input for its computation. A program can modify that input to create its output. Unix pipes was the name giver for Yahoo Pipes, an early form of visual programming. The developers at Yahoo were emulating the concepts of Unix pipes but visually, in the browser, something completely new at the time.
As a programmer, connecting pipes between programs is completely natural. As an electrician, I wire together components that create a state of existence.
Which approach is better?
Neither.
To answer this question would be to say that object oriented programming is always better than functional programming. It depends on your problem space and which approach could solve the problem the quickest, or the best, or has the best documentation, or maintainability, or costs to implement, or whatever the focus might be in solving the problem.
This comparison of flow diagrams to programming languages leaves the door open for a tool that integrates both approaches into a single tool. If only Node-RED had a state machine “flow” that could be “drawn” to track the state of the system while at the same time having a data driven, event machine.
Many different programming languages have come and gone and many will come after them, a commonality between all of them is the representation of functionality and the representation of state. This appears to be the same for flow diagrams and visual programming in general. We should not forget the learnings we have already made.
Statelessness within Node-RED
Node-RED aims to be stateless.
One of the things I learned early on was that state should be avoided in usage of nodes. Nodes should ideally be completely stateless and solely act on their input. This is possible since an external datastore can be used for state. State can be retrieved and updated as the data travels through the flow.
Ironically having an explicit state machine flow within Node-RED would isolate state to those flows. What can happen currently is that state sneaks into the nodes via storage calls to the flow state or the global state. Nodes can easily access the flow and global state objects, just as they can access the msg object and so the risk is there that state “just develops”. This state is not visually recognisable and hence hidden.
Hidden state that sneaks in through the backdoor is particularly annoying, hence a concept that puts state front and center would help to avoid the problems of sneaky state.
Colour
Noisecraft uses colours for the wires not for the ports, Blender uses colours for the connector-ports not the wires and Node-RED uses colours for nodes, the nodes representing the functionality.
Each visual programming application sets a different focus for the use of colours. In a sense colours highlights the central ideas of the developers since any developer knows that colour catches the eye, colour can warn and guide, hence colour should be used sparingly.
Why is this important?
The importance of this realisation for me is to recognise that not all flow diagrams speak the same language even if they use the same terminology.
A wire is something completely different in Node-RED and Blender or the functioning of a node is different. But these difference are not visually clear, all the viewer sees are rectangles connected with lines, all of which has colour highlights.
Programming languages can be visually differentiated but flows cannot, this is an example of flows that might well be identical but have completely different meaning depending on the approach of the engine interpreting the flow.
Post Scriptum
Node-RED can also be used for other wonderful things, for example the mindmap for this article was constructed in Node-RED. This gives Node-RED flows a further meaning: perhaps they represent a mindmap?
The grey one-or-zero concept is my description for the one or zero input ports and all ports - input and output - are grey user interface of Node-RED. ↩︎
Comments powered by giscus
The author is available for Node-RED development and consultancy.