<p>Now that Pipes got an <a href="https://pipes.digital/blog/7/Pipes%20CE:%20A%20free%20version%20of%20Pipes">open source release</a>, I’d like to present this dive into the architecture of the software. I hope it will be helpful for anyone trying to build a similar system, and especially for developers wanting to customize <a href="https://github.com/pipes-digital/pipes">Pipes CE</a>.</p>
<h3>Software overview</h3>
<p>Pipes goal is to provide infrastructure for users to work with data from the internet. It provides a visual interface where data flows from block to block and each block does one operation. Internally, each data item is an item in an RSS feed that gets sent from block object to block object, and also external data input is often an RSS feed, though it is possible to fetch data from webpages and to send XML or JSON data to webhook blocks. A typical example for what users do with Pipes is combining multiple feeds into one, filtering them, or creating feeds for sites that do not provide them. It definitely stands in the tradition of <a href="https://en.wikipedia.org/wiki/Yahoo!_Pipes">Yahoo! Pipes</a>, hence the name.</p>
<p><a href="https://www.pipes.digital/pipe-example2.png"><img src="https://www.pipes.digital/pipe-example2.png" alt="Pipes example" /></a></p>
<p>Pipes is a <a href="http://sinatrarb.com/">Ruby/Sinatra</a> application, uses <a href="https://www.sqlite.org/index.html">SQLite</a> as database engine and <a href="https://dmitrybaranovskiy.github.io/raphael/">Raphaël</a> for the main part of the frontend, the editor. Ruby works well here, since it is a very expressive language with a healthy ecosystem around XML (<a href="https://gitlab.com/yorickpeterse/oga">oga</a>, <a href="https://nokogiri.org/">nokogiri</a>) and some useful modules for RSS/ATOM normalization (<a href="https://github.com/feedparser/feedparser">feedparser</a>, <a href="https://github.com/feedjira/feedjira">feedjira</a>), and creation (<a href="https://ruby-doc.org/stdlib-2.4.0/libdoc/rss/rdoc/RSS.html">included</a>). Sinatra provides the webserver parts of the system and the HTML views, this part is rather small. SQLite is the data store for everything, including the pipes serialized to JSON and cache data. And finally Raphaël abstracts creating the SVG part of the javascript powered editor, painting the blocks and manipulating them.</p>
<p><a href="https://portier.github.io/">Portier</a> is used as the auth system. It enables logins without having to store passwords. Users log in with their email address and only have to confirm that they control that address, by either clicking on a link sent to them or, for Gmail, logging in with their Google account. That work is done by an external broker, Pipes just has to check the token sent back by that system.</p>
<h3>Backend</h3>
<p>I will start with a description of the backend, because defining the data structure is also how Pipes started.</p>
<p>It began with the realization that a pipe can be thought of as a tree. The pipe’s output object is the root of the tree, the block that connects to its input are its children, and so on, until you reach the blocks that have no inputs, the leaves. That will usually be a Feed or a Download block, though that later expanded to also allow other Pipes (as represented by one block), Twitter or the Webhook block. Running a pipe means asking the root object of the tree for output, which will ask its children for input, which iterates through the whole tree until blocks that provide input are encountered. This data then flows back through the tree up to the Output block, with each block doing its own operations on the transported data.</p>
<p>So when you look at this pipe, where two Feed blocks fetch some data, one is filtered, then both are combined into one feed and that combined feed set as pipes output:</p>
<p><a href="/blog/architecture_example_editor.png"><img src="/blog/architecture_example_editor.png" alt="pipes example in editor" /></a></p>
<p>You are actually looking at this execution tree:</p>
<p><a href="/blog/architecture_example_pipe.svg"><img src="/blog/architecture_example_pipe.svg" alt="pipes example tree" /></a></p>
<p>The code that enables this structure is minimal. There is a <code>Pipe</code> object, defined in <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/pipe.rb">pipe.rb</a>. There is also an abstract <code>Block</code> object, defined in <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/block.rb">block.rb</a>, the parent class for all blocks. Those two classes together define the pipe execution flow. An instantiated subclass of <code>Block</code> has an array of <code>inputs</code>, containing other blocks. When the system runs a stored pipe, this happens:</p>
<ol>
<li>A <code>Pipe</code> object is created</li>
<li>That object fetches the JSON structure as stored in the database and sets the output block as its <code>root</code></li>
<li>Then it creates the block objects that are the inputs of the current block, by recursively calling <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/pipe.rb#L68"><code>createInputs</code></a> on its children and their children.</li>
<li>To finally get the output, it calls <code>Pipe.run()</code>, which calls <code>Block.run()</code>, which will go through the tree as described above.</li>
<li>The actual data manipulation work is done in a <code>process</code> function each indiviudal block has to define on its own. Look for example at <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/blocks/filterblock.rb#L6">the process function</a> of the <code>FilterBlock</code>.</li>
</ol>
<p>To define pipe execution like that has a hidden advantage: When you want to see the output of a pipe at a specific node of the tree, at one specific block, all you have to do is set that block as the root element of a new <code>Pipe</code>.</p>
<p>This describes almost all classes and files of the backend. Additionally, there is a <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/server.rb">server.rb</a>, the entrypoint for the Sinatra application and where all routes as well as some helper functions are defined. <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/twitterclient.rb">twitterclient.rb</a> is a small wrapper around the <a href="https://github.com/sferik/twitter">twitter gem</a> and used by the twitter block. <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/downloader.rb">downloader.rb</a> is central for most pipes, as it is used by the Feeds and the Download block to fetch data, and the central cache is set here that prevents requesting data from other servers too often. The <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/Gemfile">Gemfile</a> describes the requirements, the <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/config.ru">config.ru</a> makes the application runnable by the usual ruby commands and also initializes the <a href="https://github.com/moneta-rb/moneta">Moneta</a> session store, that prevents users from being logged out when the server process restarts. HTML templates are stored under <a href="https://github.com/pipes-digital/pipes/tree/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/views">views/</a>, ERB is used for them. The individual blocks are stored under <a href="https://github.com/pipes-digital/pipes/tree/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/blocks">blocks/</a>. Images, Javascript, CSS etc are in <a href="https://github.com/pipes-digital/pipes/tree/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/public">public/</a>, accessible to users.</p>
<p>As you probably noted, this is all bundled into one application. The way a pipe works, it would be easy enough to divide the Sinatra part of the application and the Pipes part, into what could also be called server frontend and worker backend. That would provide a way to scale horizontally, over multiple processes and even multiple servers. So far this just has not been necessary, upgrading the server has been enough to keep pipes.digital running. But keep that option in mind if you run into performance limitations in deployments with a big amount of individual pipes.</p>
<h3>Frontend</h3>
<p>The website uses server side rendering, as provided by Sinatra. But that describes only the starting point and the less important parts of the frontend adequately. The heart is the editor that gets initialized by a HTML template (<a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/views/editor.erb">views/editor.erb</a>), but then fetches and sends data via AJAX, more akin to a Single Page Application. It is a big SVG canvas powered by Raphaël, with absolutely positioned HTML input elements mixed into it, to make the blocks configurable. Its code lives in <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/public/pipes-ui.js">public/pipes-ui.js</a>.</p>
<p>The main objects here are blocks and connectors. Connectors can be Inputs or Outputs, and there exist TextInputs and TextOutputs for the text fields that double as parameters when calling a pipe. Let’s look at the Filter block as an example.</p>
<p><img src="https://www.pipes.digital/imgs/filter_block_tiny.png" alt="Filter Block" /></p>
<p>In simplified form it is defined like this:</p>
<pre><code>function FilterBlock(x, y) {
Block.call(this, 1, 1, x, y, 'Filter', 200, 150);
this.userinputs[0].style.width = '150px';
this.userinputs[0].required = true;
this.userinputs[0].placeholder = 'keyword';
var textinput = new TextInput(x + 29, y + 37, this);
this.textinputs.push(textinput);
var field = document.createElement('select');
var all = document.createElement('option');
all.value = 'all';
all.text = 'all';
var content = document.createElement('option');
content.value = 'content';
content.text = 'item.content';
field.appendChild(all);
field.appendChild(content);
field.style.position = 'absolute';
field.setAttribute('data-xoffset', 30);
field.setAttribute('data-yoffset', 75);
field.style.left = x + 30 + 'px';
field.style.top = y + 75 + 'px';
field.style.width = '150px';
field.name = 'field';
document.querySelector('#program').appendChild(field);
this.userinputs.push(field)
}
</code></pre>
<p>What happens here? First, Block.call creates a Block object. The constructor of the Block object (that’s just the function <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/public/pipes-ui.js#L193"><code>Block</code></a>) creates the needed SVG elements. The two <code>1</code>s define the number of Inputs and Outputs the block will have, <code>x</code> and <code>y</code> are the position on the canvas, <code>'Filter'</code> the title as shown in the title bar of the block, and finally the width and height are set to <code>200</code> and <code>150</code>.</p>
<p>The Block constructor also creates a userinputs array and adds one HTML input element to it, a text box. That array is where all the inputs of a block have to be stored. That way, other code like the drag’n drop handlers know which absolutely positioned inputs belong to which block. This first userinput element is then changed, a width is set, it is marked as required and it gets a placeholder.</p>
<p>Next, it also gets a TextInput, allowing connections to TextBlocks. That’s the green circle at the left of the text input shown above.</p>
<p>The code that follows shows how to add additional userinput elements, in this case a HTML select element with two options. Note how elements like this are added to the DOM (since they are absolutely positioned and regular HTML elements) and also to the userinputs array, for the link to the block.</p>
<p>Not shown here but working similar to the userinputs and textinputs arrays is the <code>deco</code> array that does the same thing for Raphaël SVG elements, used for example to add text and lines to blocks.</p>
<p>Other main classes, but something you probably won’t need to change, is <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/public/pipes-ui.js#L1173"><code>Connector</code></a>, the parent of <code>Input</code> and <code>Output</code>, the half-circles at the left and right of input blocks. They manage the block connections.</p>
<p>When a pipe is saved these steps happen:</p>
<ol>
<li>The function <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/public/pipes-ui.js#L1551"><code>save()</code></a> is called.</li>
<li>It calls <code>serialize()</code></li>
<li><code>serialize</code> iterates through the <code>blocks</code> array and creates a JSON representation of their current state, including which blocks they connect to</li>
<li>This JSON array gets sent to the backend</li>
</ol>
<p>While I’m happy with the resulting user interface, this approach to building it involves a lot of manual coding work and is not modern. It would be quite interesting to see an implementation of this interface based for example on Vue, as long as it involves no javascript build process. Not bloating the development process with such an abomination was a high priority goal when starting the project.</p>
<h3>Database</h3>
<p>SQLite is used to store all data in a file. The file <a href="https://github.com/pipes-digital/pipes/blob/12b5e3681bb7f75788b46dc403ff79cdbcb4066e/database.rb"><code>database.rb</code></a> creates all tables and the functions for accessing them.</p>
<p>Important for understanding how Pipes works are only two: <em>users</em> and <em>pipes</em>. Let’s look at users first:</p>
<pre><code>CREATE TABLE IF NOT EXISTS users(
id INTEGER PRIMARY KEY AUTOINCREMENT,
email TEXT UNIQUE
);
</code></pre>
<p>Every user get its own id and a field for the email address. Note how that address is not the id, to make it easier to change it later.</p>
<p>The data for each pipe is stored in the second table:</p>
<pre><code>CREATE TABLE IF NOT EXISTS pipes(
id INTEGER PRIMARY KEY AUTOINCREMENT,
pipe TEXT,
title TEXT,
description TEXT,
user INTEGER,
preview TEXT,
public BOOLEAN DEFAULT 0,
publicdate INTEGER DEFAULT 0,
date INTEGER DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY(user) REFERENCES users(id)
);
</code></pre>
<p>The purpose of most of these fields should be clear. Each pipe gets its own id, can have a title and a description and is linked to a user id. For shared pipes <code>public</code> is set to 1. Most important is the <code>pipe</code> column. It contains a JSON representation of a pipe, as created in the frontend when serializing the editor content. That JSON structure is later transformed into the tree, as described above.</p>
<h3>Outlook: Open development questions</h3>
<p>The above describes Pipes as it is today. But how could future development look like?</p>
<p>Much of the development so far was user driven. Exchanges by email or in <a href="https://github.com/pipes-digital/pipes/issues">the issue tracker</a> lead to improvements, big and small, like being able to run javascript before crawling sites, create CSS selector for fetching data from sites by clicking on the target elements in an overlay, or keeping media enclosures. I’m sure more will follow, but that development is unpredictable to me.</p>
<p>One thought I return to quite often is the focus on RSS feeds. Right now, that is baked hard into the software, as the data exchange format between blocks. That plus normalization is a big part of what makes the current approach as powerful as it is. But I often wonder whether there could be a better approach, something that still allows manipulating RSS feeds and creating them as pipe output, but making it more straightforward to work with raw text, XML, JSON or other structured data instead. That was also the starting point of the <a href="https://www.pipes.digital/blog/archive/howto">usecase focused blog series</a> describing how Pipes can be used today. Finding a better concept could allow supporting more usecases (how would they look like?) and also remove the need for normalizing RSS/ATOM/JSON-Feeds before manipulating them, which could become an optional step of the feed block. One advantage there would be avoiding the issue of sometimes removing parts of the original feed not covered by that normalization.</p>
<p>Splitting the backend into a server part and a multiprocess program running the pipes would be an optimization useful for big deployments. It could also be nice when deploying to multicore servers with weak cores, like those ARM servers scaleway used to offer or a Raspberry Pi at home. <a href="https://github.com/resque/resque">Resque</a> could work well for that.</p>
<p>Porting the user interface to a more modern javascript library could be worthwhile if it would result in new capabilities for the user, maybe even supporting new usecases that way.</p>
<hr />
<p>Questions, suggestions? Join the <a href="https://gitter.im/pipes-digital/community">pipes gitter channel</a> or <a href="https://github.com/pipes-digital/pipes/issues">open a github issue</a>.</p>