It now has been two years since I started working with Node-RED and I continue to use it everyday. For me, Node-RED shines in its simplicity, its extensibility and adaptability to a variety of tasks.
My personal aim remains to create visual tooling around Node-RED to support visual development using Node-RED. Perhaps allowing visual programming to become comparable to textual programming and thus creating an alternative to one dimensional textual programming. The unfortunate truth is that visual programming remains faraway from becoming a mainstream alternative to the one-dimensional textual programming.
My favourite comparison has become punchcards: no one continues to use punchcards to program a computer, why do keyboards remain the main tool for coding today? Why is there not a development to find alternatives for programming?
The upsurge of AI and the belief that it will provide solid, functioning, stable code has not helped. The future might well be: no one uses a keyboard to code, everyone uses voice over AI to code their favourite devices. It appears many believe that, comparing forum user counts bears that out: OpenAI has 800k+ users, Node-RED 21k+. (Discourse forums at time of writing.)
Anyway, following on from one year Node-RED, this is then my two years Node-RED retrospective.
FlowHub.org has become an attempt to provide a simple visual-first code hosting service. FlowHub provides a visual layer on top of GitHub repos, providing a visualisation of Node-RED flow code hosted at GitHub.
FlowHub became more usable with the introduction of GitHub personal tokens. The idea is that tokens allow FlowHub to commit changes to specific user defined repositories. I maintain several GitHub repos via FlowHub, both public and private.
I also added oEmbed support so that a FlowHub link can be embedded in third party web pages. Unfortunately the oEmbed standard (and idea) is going the way of the Dodo because of the presumed potential harm that third-party content might cause.
LinkedIn allows FAANGs to embed their content in posts. If you believe the hype then it would be a possible for others also. The truth is that the people responsible for oEmbed at LinkedIn have left and no one seems to have taken over the responsibility for adding new content providers. Unfortunately this situation is another example of wall-gardens occupied by selfish Giants.
I actively tried to get on the oEmbed list but no one at LinkedIn seems to know how, so FlowHub oEmbed continues to wait for its day.
Node-RED is a kernel (think Linux kernel) ideally so that most of the extra functionality (if not all) should be provided via nodes, plugins or sidebars packages.
I explained this idea in the forum. Why do I believe this?
Linux (or Unix) development has always been concerned with defining clear interfaces which provide a set of solid APIs upon which all tooling can build. This makes for clear demarcation: Linux is responsible for managing hardware interrupts, memory, CPU and providing a standardised path for others to interact with whatever hardware is available. I see Node-RED doing exactly the same, especially since the package management (in form of the palette manager) is done so well, also some demarcation already exists as well defined internal APIs.
Perhaps what I would like to imagine is that this idea would be more broadly accepted so that development via custom node packages can be seen as a first-class citizen and not something that is rediscussed every-time a node package does something that - potentially - Node-RED should be doing directly.
A good example of this discussion is the prototype for a dynamic debugger for Node-RED. Following up from a short discussion on Slack and then a longer discussion on the forum, I created a prototype for a more general and holistic Node-RED debugger. After much iteration, I finally came up with a version that is mostly “safe” to use.
Initially it was tracing all messages, but that was indeed too much for the editor. I refactored the feature to use a selection of nodes by default. Complete tracing of all messages is still possible but requires a confirmation checkbox.
Some would have liked to see this in the Node-RED debugger project but I consider what I have done to be a prototype that can be tested and developed further before a discussion of integrating that into a Node-RED debugger.
I still don’t quite understand why this is an issue. After all Node-RED has a wonderful plugin architecture similar to the Linux Kernel and the Unix operating system (think of Unix command line tools as plugins), so creating extensions for Node-RED should be as simple as possible and encouraged. If some feature developed via a plugin might be integrated into core Node-RED, then fine.
Why do I think this? My experience of coming up with the ExecWithStdin node. I have no problem with creating a pull request provided I do not need to depend on that being merged to utilise that feature. And that is the case since I have a custom node that does exactly what the ExecWithStdin node does. I can move on and do other activities instead of trying to push through a PR that no one really wants. I can still use the feature and others can also install the custom node if they desire that feature. And if no one wants it, I do not mind since I have not wasted my time pushing a PR that no one really wants but I need.
Even though I created the first outside of Node-RED visualisation of flows package, I wanted to have an interactive in-browser Node-RED experience for my flows hosted at FlowHub. I came up with the serverless Node-RED in-browser experience with the codebase at GitHub. Only later did I realise that there have been other attempts.
The difference is that there is no server and all flow execution is simulated in the browser. Most nodes do not work and the Node-RED experience is limited however it does provides a taste of using the Node-RED editor without having to install the Node-RED server. That is the intended advantage of having a serverless Node-RED instance.
I integrated the serverless Node-RED with FlowHub, it is possible to open every flow hosted at FlowHub in the serverless Node-RED instance and therefore partially execute flows in the browser. I can even edit flows using my FlowHub tokens to commit changes to GitHub via FlowHub.
Forum post discussing the serverless Node-RED.
For reasons that will remain lost in the mists of time, I needed a form of visual obfuscation. First question: what does that even mean in a visual programming environment? We all know that Javascript and CSS code can be minified and uglified, thus making the code computer-only understandable - can that be done with Node-RED flows?
It can and in combination with minifying the code behind the nodes, it can go a long way to making visual code un-understandable for humans. It might well have a limited use cases, however if flows were sold in some kind of marketplace, then this could a method for providing samples.
I did use this feature for a specific project, so the work wasn’t all academic. Also it does provide one of those tools that is common in textual programming. After all, is not compiling code a form of obfuscation so that only computers can understand that which we have written?
This was some work that Bart and I did together and something that ruined the obfuscation - what is the point of putting all nodes on top of one another when there is an auto layout tool to separate them out again?
But in fact, auto layout is something that is not that useful since only a human can bring meaning to the layout of a flow, a computer can make it all straight and aligned, but that does not add semantic meaning to the flow - syntactical meaning yes, semantic no. Perhaps this would be a good case for AI in visual coding: Dave: HAL align and layout this flow so that a human can understand it. HAL: Sorry Dave I can’t do that.
So for me, autolayout is something like auto indent in a textual editor. Proper indenting makes code more readable (and perhaps executable in the case of Python) but it does not - generally - add to the underlying understanding of the code. That is, the algorithms used in the code are not made more obvious via automatic indentation.
What I discovered for me is that I enjoy creating flows that look like what they do, for example a decision tree or an LED display, or even message loops with exit conditions. Here visual coding has an advantage over textual programming, textual programming rarely visually appears as the algorithms being implemented.
Orchestration using Node-RED was also something I thought about. For this purpose, I create an example ETL pipeline for ingesting large datasets. This is the purpose of the streaming nodes: to break large datasets into chunks that can be piped through Node-RED. Due to Node-REDs nature, it is not a good idea to pass large messages through flows.
The interesting feature of the streaming nodes is that they become a meta-flow: a flow that describes the flow of data once it has been executed. A typical streaming flow needs to define where the streaming data should go during an initial execution and then it executes that description. This makes Node-RED to an orchestration tool: describing the flow data not executing the flow.
Orchestration is about visually describing the flow of data outside of Node-RED by using Node-RED. Node-RED becomes a tool for describing data flows outside of Node-RED, a meta-flow engine.
Sometime in April a representative of FlowFuse contacted me to ask me whether I would change the license of my flowviewer package so that FlowFuse could integrate that software. Strangely enough I said no.
Why? To clarify why I use the Don’t Do Evil license: I use it to make a statement concerning humans ever growing need to be evil. Be it in name of a country or a company, being evil is simply simpler than being friendly. I definitely don’t expect it to be enforceable nor prohibitive, its a statement about a society that doesn’t not ask itself what is evil and why am I being evil?
Ironically this what ZJ from FlowFuse actually complained about - what is evil? It is strange that it seems hard to define what is evil since we have morals, we have laws and we have the ten commandments, yet we live in a society where “what is evil” is a question too far.
This is all I desire, that people ask themselves what is evil. Whether you copy my code or whether you obtain a copy via AI, I doubt very much that I will have the financial resources to enforce something that is not enforceable in our, ironically, humane and just world.
But why use a license text to transport societal messages? Because, it seems, that the only thing people read when using Open Source Software is the license. Which is sad. The time wasted to discuss and overcome legal aspects of programming has become enormous and the secret truth is that large corporation do not even care. (Remember: AWS and Elastic.)
That was my second year involved in Node-RED and it was a year filled with realisation that everyone just wants to use AI to do everything. Making visual flow based programming popular has become that much harder. As Human development drifts towards ephemeralization, a technology that requires a steep learning curve, promising efficiency later is a difficult sell. Particularly compared to a technology that provides 90% of the solution in a matter of seconds - surely the rest is only a matter of minutes away!
I don’t want to sound too pessimistic, I still really enjoy using Node-RED. I can even see a 3D future where code is created in a three dimensional editor. After all, is anyone still using punchcards to program computers? So why do we believe that the keyboard is the be-all and end-all of how to program?
For me, Flow Based Programming is valid approach to coding and it should be explored and developed. Especially since we have the computation possibilities to fully use visual flow based programming. Keyboards only became popular because the hardware could support them. For a long time, visual programming was not seriously possible because our displays could not support sensible GUIs - I remember using B/W Sun monitors and being amazed about the graphics! But that has changed and we should recognise that change.