…(radio silence)… (static)… and we’re back, with lots of news, assets and experience under our belts.

It has been two years already, and in this time span we have developed an incredible amount of open-source technology (somewhere between 250.000 and 300.000 lines of code, which regardless of what these metrics may mean to you, is nevertheless a badge of the massive amount of work we have put into it) which has been until now, for various reasons, mostly dormant… and now the sleeping giant is waking up.

A Journey of a Thousand Miles Begins With a Single Step

Two years ago, there was a “itsy-bitsy” technical problem needing to be solved: migrating data from a legacy information system to a new one being developed by us. The solution was simple and straightforward, and its prescription was: understand the source data model, pick up a bunch of nails, and “hammer-up” a script to copy the data to the new one.

“But why?” — asked the relentless blissfully ignorant kid within us. Why go through all the trouble just for this one single migration? Why not re-use it in potential future migrations for other deployments of the same system?

Be it not for the miscellanea of customizations found in each deployment, this would be trivial. Each installation would require a new, but similar, migration script. Using a traditional development approach in order to solve this problem would be the same as shooting yourself in the foot, we would be wandering in spaghetti code land in no time. This problem was crying out for modularity, a concept we had learned back in college, but which had found its way to the back of our heads, not receiving the attention it probably deserved.

Hence came about the inevitable dive and swim through the somewhat muddy waters of different technologies, paradigms, specifications, concepts, languages and all-things modularity related, from which OSGi stood apart as a major inspiration.

An UnHealthy Obsession With Modularity

Call it proactive laziness, or an avant-garde distaste for unnecessary complexity, but in the end, we set out to build our own modularity platform capable of providing us the code decoupling ease we required, while throwing away the behemoth that was the overhead of building modular applications with the platforms and accompanying resources available at that time.

About four thousand lines of Python code later, there was file called “plugin_system.py” that took care of supporting the solution to the migration problem by allowing us to create a basic migration plugin, and later extending it with the additional logic necessary to migrate from installations with different customizations, without ever having to touch the original code.

Then we created a console for that plugin system, and that was a plugin too. Then that console needed new commands, so we created new plugins to provide those commands. Then we wanted to access the console remotely, so we created a XMPP plugin, fed it the console plugin, and fooled around with the plugin system by issuing it commands via Gmail chat.

This plugin development frenzy grew completely out of proportion. Plugins forced one to separate concerns, so each piece of code had a beginning, a middle and an end, instead of being an ever-expanding intertwine of different features whose development seems never-ending. The cherry on the top was that new and unexpected uses for previously developed plugins kept popping up, which made the development seem to have a cumulative effect regarding its end results.

Reinventing Improving the Wheel

We were using a traditional JBoss stack to support the information system we were developing, while the ecosystem around the once lonely plugin system kept growing both in power and responsibility. Until one fateful summer day, a friend visited our headquarters, saw the project and asked why the hell we were using JBoss and not our plugin system… we had no real answer for that one…

Long story short, we scrapped JBoss and moved everything to the plugin system, which meant replacing all the Java technologies we used in JBoss with their Python equivalents. The plugin system received another level of attention and started growing beyond anything previously imagined. Replacing existing technologies with home-made plugins that addressed the same problems became a trend by the way. While at first glance this move may seem like reinventing the wheel, and it was… our previous wheels were not modular, and these were.

The plugin system stopped looking like a script, but more like a fully fledged development platform, due to the wide scope and number of the plugins we developed for it, which ranged all the way from an ORM to an HTTP server, slowly becoming a fully home-brewed enterprise stack.

We had to give it a name, and after many not too bright attempts, we arrived at the name Colony, which may have seemed mildly lame at first, but now we wouldn’t call it anything else.

The Tao of Modularity

Every software project seems to grow quite fast in the beginning, and after a while stale its progress, due not only to the amount of code surpassing the amount of eyes available to read it, but mostly due to the intertwining of unrelated concerns that eventually finds its way into a solid architecture, mostly due to unexpected changes, melding it into a plate of spaghetti, making the small amount of eyes available still have to look at all code, therefore making further developments unsustainable.

With the modularity paradigm supported and enforced by Colony, growth always seemed sustainable, because it was much simpler to devise architectures that were future-proof by following this Tao of programming. Designing plugins strongly encouraged thinking of their interface first and code their internals later, promoting a higher depth of thought that would later reveal itself as a high resilience to change, and a higher probability and ease of re-utilization in different contexts.

Quest for the Holy ERP

Megalomania set in again, and the humble information system project morphed into a quest for the Software-as-a-Service Enterprise Resource Planning System-to-Rule-them-All that would solve all problems harassing users of the ERP giants dominating the market: high acquisition, maintenance and customization costs, and low accessibility, amongst others.

Colony had to step up to this new goal, and so it did. If Omni was going to live in the browser, we needed not only modularity on the server-side, for the ERP’s data and logic, but also in the browser, to make the interface adaptable to each user and easily extendable through time, without ever needing to reload the browser page that hosted the application. We did this by porting Colony to Javascript so it could run in the browser, and we nicknamed this implementation Colony Web.

Books shouldn’t be judged by their cover, but traditional ERP interfaces were… let’s say… ugly. We wanted to make ours lively, snazzy and full of life, so we looked for a Javascript widget library that would help us accomplish that, and found our way to ExtJS, which was awesome, and was part of Omni for a long time… but after diving too much into its core in order to customize it to our needs, we decided that it was becoming too much of an overhead, so we built our own widget library, which we call Colony Web UI.

In the end, this quest brought forth a flourishing garden of new plugins for all purposes and tastes, ultimately spawning Omni, our flagship Software-as-a-Service/Cloud Computing service. A web service that can load/unload plugins on the fly without ever reloading the container browser page, and therefore a great target host for a new era of rich internet applications, making it pretty much like an operating system for the web (a rather crass metaphor), and leaving the information system itself as just a set of plugins that can be loaded in Omni like any other plugin bundle.

Now What?

At this point there’s not much left technology-wise that isn’t fully under our control. Omni is out in the wild and working successfully, we are currently smoothing the edges of the current features and of Colony as a consequence.

Meanwhile, we started offering consulting services for potential clients looking to have interesting problems solved for a fair price, with our focus being on SaaS, Social Networking, Mobile, and Cloud Computing, as these are areas we have been in the trenches with throughout these years.