Click to close
When I started out working with Node-RED, I was overjoyed to have found a universal, well thought out visual programming environment. It became possible to bring my visual mental models of algorithmic code to the fore. I found myself creating beautiful visual representation of algorithms and code. I began doing everything with Node-RED, any small task that I would have normally created using a textual programming language became a visual collection of rectangles and lines.
As I created more and more flows, I realised that there were no simple mechanism to organise my visual code. I was starting to get confused and lost amongst my code diagrams. Too much beauty, too many themes, not enough organisational clarity.
Drawing parallels between visual programming and textual programming, I realised that textual programming has been improved by the tooling that organises the code: textual programming has developed organisational clarity. Why does visual programming not have similar visual tooling for that organisational clarity?
Organisational[1] clarity is exemplified by software configuration management of textual programming. This encapsulates version control tools such as Git, Subversion, CVS or RCS. Using version control to maintain the past has become essential to textual programming.
I thought about the tooling that would be required for visual programming. It could not simply be a one-to-one copy of the tooling available for textual programming. This would mean losing the visual aspects of code. How then to go about this? Having no idea of the how, I simple began to experiment. I created a basic flow json to SVG visual transformation to explain the workings of visual programming, specifically Node-RED flows. This gave me an approach to explain the now but what about the past? After all, being able to compare the past with the present provides learnings for the future.
This then became the second component: version control. For this, I used GitHub as the store for my flows. This gave me version control on textual basis, all flows files just being json files. All I had to do was generate the visuals of various versions and compare these. This became a matter of SVG layering and using opacity to blend between versions.
Being able to compare my documentation of the present with the visual past improved my visual coding. I became more certain of what I was doing, what had changed, what had improved and what was missing. This gave me the freedom to experiment on my flows but with a safety net.
Without realising it, I had translated the concepts that I knew well from textual programming into a visual representation. I could maintain versions of my code and compare those visually. More importantly I integrated my comparison of visual code into Node-RED, I no longer needed to rely on a textual interface.
All this eventually became FlowHub.org which represents a visual platform for documenting and versioning visual code. The idea was to have a gist platform for visual code. Each flow is a gist of code that could be combined to form larger codebases comprise multiple flows.
The Node-RED client is not an code editor, it is a flow editor. The client describes the workings of the Node-RED server. But what happens if the Node-RED client was an code editor with a builtin server? What if my flows described code that should be installed on the server?
Extending the functionality of Node-RED with custom nodes, I came into contact with the build and deployment process of custom nodes. I soon came to dislike the go to textual editor, make change, restart Node-RED, back to textual editor, repeat cycle of custom node development. I wanted to stay in Node-RED when I create my custom nodes. But how could this be done? With custom nodes of course!
The first time I opened Node-RED, I saw rectangles connected via lines. Before I came to realise that the lines represented data flows and the nodes computational units, I saw shapes. A shape can be anything. So I decided that a node could also represent a file in node package. What if my node package for my custom node would be represented by a flow inside of Node-RED?
I began to experimenting with how to represent different file types in Node-RED. How to represent a directory structure when nodes are flatly connected? A simple solution is to give each node a filename and path in the package. The PkgFile node became an extension of the template node with a path and filename specification.
The next step was to create a tarball from a collection of PkgFile nodes. For this I created the Tarball node for packaging a collection of PkgFile nodes into a single tgz. The next step was to install that tgz into Node-RED and have my custom node installed into Node-RED without leaving Node-RED. This became the NRInstall node and the whole collection of nodes became the NodeDev package.
I could now develop, build and install my custom nodes within Node-RED. No more switching between my text editor and Node-RED, no more terminal for restarting Node-RED - custom node development became holistically doable in Node-RED. Reloading Node-RED became an exec node that restarts the server. Node-RED became my build server and my code editor.
Since my node packages became flows, I could use FlowHub.org as a visual version management tool. Hence the NodeDev flow is the code that generates the NodeDev package.
The flow has functionality to create the tgz tarball, to install that tarball into the Node-RED server, to push my changes to the GitHub repository and finally publish the package to NPMjs.com. I now maintain all my Node-RED packages in this way. A flow based presentation of node packages actually makes GitHub secondary since I can uglify my JS code before committing to GitHub: only the flow contains the real code.
Node-RED is well built software, I realised this when I discovered that the entire editor is just one giant SVG image with a lot of jQuery code on top of it. Many would be surprised to combine the works well-built and jQuery together in one sentence. Why not React or some other “modern” Javascript framework? I would put it in one word: complexity.
Node-RED is easy to extend because the underlying technology is simple and easily understood. Many frameworks are built around the MVC (Model View Controller) pattern but each has their own interpretation of that pattern. It becomes difficult to differentiate and understand. jQuery is simply a unified interface to Web concepts such as CSS, JS, DOM and HTML, it has no underlying MVC framework to understand.
SVG is a standardised XML representation of vector graphics.
Both SVG and jQuery are old technologies hence both are proven, reliable and well implemented by all browsers. On the basis of those two technologies, the developers of Node-RED added an abstraction layer for representing SVG bezier curves as data flow lines and SVG rectangles as nodes for computational units applied to data.
Additionally many UI actions are encapsulated in higher level concepts such as highlighting or selecting nodes, marking flows as having changed, redrawing flow workspaces or zooming the view to a specific node. Taken together, this makes extending the Node-RED client simple[2] since one can access the “raw” html using jQuery but at the same time use flow concepts to manipulate the Node-RED client.
This is one of the nicest things about Node-RED: it easy to extend Node-RED without extending the codebase of Node-RED. I can add functionality to both the server and the client of Node-RED by using packages. This encapsulates functionality to specific packages and the developers of Node-RED need not maintain features that they might not even use. All my extensions are designed for me and maintained by me (or not maintained). If someone wishes to use these, then they can install any of my packages into Node-RED.
My personal mantra when it comes to extending Node-RED is to make the Node-RED codebase more extendable instead of extending the codebase. Meaning that I believe Node-RED should offer more events, hooks and actions onto which node package can piggyback rather than extending the functionality of Node-RED itself, thus avoiding feature bloat and feature creep. Features can then be built as packages that can (or cannot) be installed into the editor by individual users. Node-RED has a superb package management, why not utilise it more?
An example of this is the visual link between a link-call node and its corresponding link-in node. Unlike the visual link between link-in and link-out node, there is no easy navigable visual cue between link-call nodes and corresponding link-in nodes. My attempt at creating one is now part of the introspection package. I used a green dot as navigational aid[3]:
I do not expect everyone to be happy with a green dot and neither would I want to add that feature, in that form, added to Node-RED. I have found my solution which is packaged in a node package that I and everyone else can install into Node-RED. I can also define a shortcut, so that I don’t have to use the sidebar. That is another great feature of Node-RED: I can define an action and then bind that action to a keystroke. Being able to extend Node-RED to the level of creating actions for keystrokes is wonderful, it provides Node-RED developers with many freedoms - both those that develop Node-RED and those that develop node packages for Node-RED.
Taken together, FlowHub.org, NodeDev and Introspection are my approach to making Node-RED a more holistic development environment. In combination, they save me much time and effort by providing shortcuts, helpers and a development process to make my Node-RED development more comprehensible and consistent.
Organisational is meant here as in organising stuff, not in the sense of large corporate organisation. ↩︎
To be honest, simple for someone who has been coding websites for years. In comparison, extending Emacs is far simpler if one understands Lisp and the concepts of list processing. ↩︎
Feature can also be tested in the serverless crash-test dummy Node-RED ↩︎