Learnings from creating Erlang-Red and how these could be applied to creating a Rust-Red. Rust-Red is a Rust backend for the Node-RED flow editor, to be able to create flow-based code visually in Rust.
The following is a list ordered by how I did this for Erlang-Red. There might be a better approach in re-implementing Node-RED for another programming language, this approach worked for me implementing Erlang-Red - your mileage will vary.
Web-server is required to handle the static content of the flow editor. There are simple routes that deliver the content and routes that support the deploy button. What static content defines the flow editor? I worked on a static-in-the-browser implementation of Node-RED a while back and for that, I created a script to retrieve everything that is necessary to re-create the Node-RED flow-editor in the browser.
Web-socket to ensure that the flow editor believes that the server to be alive and kicking. This is a simple JSON heartbeat that is sent back and forth between server and client. Else the flow editor constantly complains about the server not being available.
JSON parser for encoding and decoding flow data. There was an interesting issue though: Erlang tuples, process IDs and references. Since internal data structures can be displayed in the debug panel of the flow editor, tuples have to be JSON serialisable. Not all Erlang types can automatically be serialisable to JSON - for example, tuples, references and process IDs all have no comparable types in JSON. The solution is to convert types before sending these over the wire to the flow editor. This of course means that the type information is lost, at the moment this is a non-issue.
Add some nodes. Start with inject and debug - these allow very simple interaction between the flow editor and server. The initial steps in organising/designing the communication channels between server and editor. Something that I discovered was the importance of separation of client concerns. Here the clients are users of the flow editor executing the same flow. A single server can have many clients and each client needs to have their own representation of flow data and flow execution. This is a difficulty that Node-RED has not yet solved — there the assumption is that one server, one client, one flow, one execution. Which I can definitely understand since its a much simpler solution. In implementing Erlang-Red, I approached this problem by scoping everything using the web-socket. A web-socket represents a client. This works as far as I have tested it.
In creating the inject and debug nodes, the communication via the web-socket needs implementation. Also the data objects passed over this channel need to mimicked. This includes the inject object that is sent to the server and the debug object that is sent back to the editor to be displayed in the debug panel. Why are these data objects being mimicked? Because my intention was to make minimal (ideally none) changes to the flow editor. I ended up making a number of changes to the flow editor but my intentions were good.
In no particular order, the development of other core nodes should be done. Useful for this is the repository of visual flow-based unit tests that test internal Node-RED node functionality. Starting with the change and switch nodes is a good step because these provide traffic/message routing abilities. These then make flows look more interesting. Function and exec nodes extend the capabilities of the overall implementation, so these too become useful to implement early.
In implementing the function node, the editor (Monaco not flow editor) needs to be extended to support syntax highlighting of the underlying programming language. There is little point in having a Javascript function node for an Erlang (for example) server. So the function node must become an Erlang function node. Function nodes can be very useful since offer a method of executing native code within the framework of a flow-based program.
JSONata needs implementing for whatever the server language is. JSONata as a language, is poorly defined having a parser hand-coded in Javascript instead of having a lex and yacc definition. JSONata is ideal for such a parser since it is well-defined transformational language used to implement the most common functionality within Node-RED. Without JSONata, the usage of function nodes would be much more common, unnecessarily cluttering flows. JSONata is not a fundamental feature for flow based programming, however it speeds up development a great deal by enabling shortcuts. Functional shortcuts for example is being able to sort data without resorting to a function node. As reference, I created the Erlang JSONata parser that transformed JSONata stanzas into Erlang code. The Erlang code is then compiled at runtime. My thinking was that it would be simpler to transform JSONata to Erlang rather than creating an Erlang interpreter for JSONata.
Development language specific features can then be developed. Each programming language has its own set of specialised feature. For example Erlang has behaviours that provide constructs for creating complex architectures. These features should also be represented in flow editor so that a new set of users also feel at “home”. Implementing such features provides bridges between developer communities. Erlang-Red has the supervisor node that provides a visual implementation of the supervisor behaviour. This allows nodes to be restarted by supervisor nodes providing overall flow stability in case of failures. This feature is unique to Erlang-Red but there is nothing stopping someone from taking that idea and porting it over to Node-RED. This form of cross-pollination of ideas is something that only a visual programming language can make possible.
Map/Hash/Object access using Javascript notation. Javascript has a very liberal approach to accessing object attributes. There is the classic dot-notation, then there is the named array notation and since double and single quotes are allowed, the array notation has two different forms. These forms are allowed to be mixed-and-matched according to taste: msg.payload['var1'].var2["var3"] is perfectly legal access to the nested attribute var3. I ended up created a simple Erlang parser for this for this access notation. This made me also rethink about atom versus binary key names for maps - I ended up flipping all my code to using binary keynames.
Document what you have done. Explaining the implementation can be helpful in refactoring the software architecture. Also the usage and knowledge of Node-RED is a prerequisite for understanding any cross-over implementation. Documentation can be done in Erlang-Red, using mermaid support and the fantastic preview mode of Node-RED. I create my mermaid flowcharts using a Node-RED plugin. The documentation step is towards the bottom of this list because documenting too early risks redoing it - as I will with my documentation.
Dog-fooding from the start - Node-RED is a complete development environment in the browser, many features of Node-RED do not require a server backend. I use Erlang-Red as much as possible so that I can iron out the kinks and get a feeling for what is required. When I encounter something that I can do in Node-RED but cannot in Erlang-Red, I attempt to implement it in Erlang-Red or make a note of it. Much development work for Erlang-Red originated in me wanting to do something in Erlang-Red that I could do in Node-RED.
One final remark would be that using existing libraries to implement functionality is important. For many requirements there will be a library that implements said feature. Being able to stand on the shoulders of giants makes for a better view from the top.
For me this question does not exist - I have been working with Node-RED for a long time and have come to understand its workings. I believe that Node-RED provides the best trade-off between granularity and features for a programmer visually developing flow-based code.
As I wrote, using Node-RED has a number of advantages:
Node-RED is open source software with a good track record of being backward compatibility: flow code from the earliest days can still be imported and used in todays Node-RED.
Node-RED has good APIs for controlling the flow editor. Also the flow editor was not built, it was designed: features are well thought out and harmonise with one another instead of being “just added”. Its UI is very consistent and simple to extend - given enough knowledge. Then again, it is simple to build an atom reactor given enough knowledge. So this last point isn’t surprising.
Finally, Node-RED flow editor runs in the browser and uses simple and standardised jQuery as its web framework. That makes it compatible with many devices. The codebase is stable - no constant updating of the underlying JS framework. There is a in-browser, no server required version of Node-RED for those who are unfamiliar with Node-RED and want to get a feeling for it.
I see Node-RED as providing a good user experience and user interface for presenting the ideas of flow based programming visually.
I personally use Node-RED for my blogging software (this page is hosted on a Node-RED server), as an API backend for an application (FlowHub.org is implemented in Node-RED) and as a mind-map tool (I created this document in my mind map and then transferred it to my blogging tool - all via the export functionality of Node-RED).
My interesting is developing Erlang-Red and using Node-RED is communication amongst folks. Having worked in the IT industry for much of my life, and having been confronted with folks not understanding what I do, I was looking for a better representation of the code I create. This makes communicating with stakeholders simpler. Being able to convey understanding using pictures versus text has been clear since the thousand word picture.
Being able to design my flows in one “programming language” but having that flow code being executed in another “programming language” is another step. I doubt it will get anywhere near to that goal, Artificial Intelligence “seems” to provide the same functionality textually and far faster and today, not tomorrow.
One of the main consequences of the lack of understanding between techies who code the solution and folks who sell that solution is Artificial Intelligence - it is a search for understanding where communication has failed.
This recipe was created while describing the start up phase of flows in Erlang-Red. That description provides more technical details of Erlang-Red. It highlighted for me the importance of a proper software architecture for such a comprehensive project.