Brand New Project – “From Scratch” or “Not from Scratch?” That is the Question

Kirjoitettu - Viimeisin muokkaus

Brand New Project – “From Scratch” or “Not from Scratch?” That is the Question


When starting a new web project you need to make a decision: customize an existing open source project (make it suitable for your needs), or start a brand new project from scratch. And I assume there is something similar to your idea in open source.

Set aside business requirements – we're only discussing the technical side here.

First of all, you, as the project founder need to have a clear vision of what exactly your yet non-existent project will be like: feature set, who will use it, how it will gonna look, and how it may evolve.

You have to define a set of features - what features are needed, what features would be nice to have, and, of course, leave some space for unexpected features.

Once the answers to these questions have been determined, you can assess the technical requirements of the project in terms of performance/capacity.

And once the feature set and capacity are determined, you can finally choose a development platform. What are the general performance/capacity requirements?

Here I will mention a few common and important performance/capacity metrics for web applications. The feature set is kind of self-explanatory, but scaling issues might be overlooked, with some hard-to-fix consequences.

* HTTP requests per second - should you use a load balancer?If your project requires a load balancer, you’d better take that into account when selecting a development platform. Introducing LB into an application in further stages might be challenging and quite frustrating: you might face system core issues, such as cache fragmentation, lost sessions, etc.

* database hits per second - how many reads and how many writes?It's all about the database architecture. Whether it should be a master-slave relational database, or should it be a multi-master setup, or maybe it better suited for NoSQL or a graph database.

* bandwidth/storage – will your app use a large number of BLOBs such as images, videos, sounds? How many? Do you need a CDN? Should your web server support streaming? You may need a storage solution.

With that being said, this is kind of a speculative question - it’s hard to estimate how your project will evolve. It can remain unused by anyone. And it sucks - you’ve allocated a lot of resources and a lot of work hours for a new Twitter, but ended up with some kind of Shitter, when no one knows about it, and your project uses only about 1% of your allocated resources.

On the other hand, if your project can't scale well, or only scales up to a certain boundary, you may face another problem: if your project gains popularity, it will be difficult for you to scale, or you may even reach a dead-end (upper limit of scaling capacity).

Your answer to the scaling question is a clear bet. The closer to the predicted value – the better.

software architecture

3 strategies for raising a new project

Whichever path you choose, you will still come to one of three options:

1) you will implement your solution entirely on the existing project basis

2) you will write your own “Twitter successor” from scratch

3) some realms of your “Jiggle” (as we will call your hypothetical Google replacement) will be written from scratch, and some will be based on the existing product.

A few words to describe each of the above three strategies + some examples

1) – basing on an existing solution

Open source projects in most cases have extendable architecture, with lots of existing plugins and a well-defined and documented API for developing new ones.For instance, Joomla has 6.097 plugins (at least at the time of this post writing) in every conceivable category. Most of them are free, and some of them cost a couple of dollars.Sometimes, the product you need exists, but only in a paid form. There is no free alternative. Author's (mine) advice: if you think that $1000 or $2000 is expensive, and you are determined to develop from scratch – you are making a huge mistake. You’d better buy a ready-made solution, if you can find one. The “from scratch” way is the hardest one, which I will explain a little further. 

When choosing this path you may also need to write some code. It could be as well as just a couple of code lines, or turn out to be a massive effort put into the custom plugin development. Remember: CMS/Forum/Ecommerce etc. platforms are not perfect for development. When developing a plugin you won’t have your favorite ORM, won’t have your favorite templating system and you have to consider the platform's ecosystem – all this may slow down your development. 

2) – everything from scratch, “The Choice of The Samurai”.

 Well, it's quite self-explanatory… You open your favorite IDE, and start developing your Jiggle from zero. “Simple as”. Good luck. 

3) – a mixture of the previous 2 strategies. 

For instance, you might write a brand new search engine Jiggle from scratch, and add a phpBB forum to it, while there is some kind of integration (bridge, auth service, etc...) between the systems. 

There is also slightly another path worth mentioning: you may start with (1) to see how things will go.In that case, your project may have some rough edges, but this is the easiest and fastest way to “get up and running”. If and when you see your project gaining popularity, you may decide to switch strategy to #2 or #3. This method follows from the previous one (3), so I won’t give it a number.  

Pros and cons 

1) – basing on an existing solution

There are open-source or off-the-shelf products for almost everything today. If you managed to find a solution that fits your needs – just go for it! Customizing and extending an existing working solution is much easier than creating a new one. At least in most cases. 

In order to see when this might not be appropriate, let's just examine a real-life case, with an adult website developed by the author. I sure assumed that WordPress or Drupal would do the job, but a couple of years ago I decided to build it from scratch. This is the outline of how I arrived at this decision: 

* I am not very good at PHP

* CMS is only a small fraction of the functionality I need, I have to write a lot of code, so choosing WordPress or Drupal is rather a liability than a benefit. I need an ORM and a handy unit testing ecosystem, not an admin dashboard.

* I need a user dashboard as a single page application (JavaScript)

* Security considerations. If you’ve ever administered a web server, you probably know that literally everything that exposed to the web is constantly under the relentless gaze: scanners are checking your software and version, script kiddies running endless exploits against your site, spiders are crawling, search engines are indexing, hackers are testing, AAAAAAAAAAAAA!!!! I will sleep better, when I know that no one can see the source code of my app. 

Some other considerations:

How exactly do you work with mobile web? You may need and want to develop apps and/or separate mobile sites. But many good open source solutions are out of date now.They were “born” when the mobile web was less important than it is today, and more adapted for the desktop. 

How about scaling? If you are sure that you will need to scale out, you’d better check your selected solution – whether it supports this, and how much it will cost to adjust it for your needs.  

2) – everything from scratch

If you have never done this before, you’ll probably be surprised how hard it is, and how much time it takes. I mean the whole way, until you get your first user subscription. You have to handle all the aspects of web application: database, app layer, web design, client-side programming, and DevOps. If you are working solo, you’d better be one-man-army to pull this off. But it gives you the absolute freedom to choose your database, programming language(s), programming frameworks, and web server solutions. And, in case that you are a programmer – this will give you the ultimate pleasure. But if you going to maintain your project (DevOps + bug fixing + new features), it will give a serious headache as well... Author's (mine) advice – follow this route only when absolutely necessary. This is a really (really) hard and long journey. 

3) – a mixture of 1 and 2

It’s so flexible, that I can't even think about any cons. If you can implement it – just go for it!  


You start by defining your features set – “must have now”, “will do later”, and “might be done later”. Your next step is to estimate the number of your app users, and based on this estimate you will calculate the required resources and capacities, such as memory, disc space, bandwidth, database type and architecture. Armed with this knowledge, you can and should(!) check existing solutions. When found, you might evaluate them and even do some POC. If you’ve found an open-source solution that suits your needs – hooray! If not – well, then a long and harsh path of a samurai awaits you.


Seuraava artikkeli

What is Customer Experience Design?