This Website is Weird

I can honestly say this is the weirdest website I’ve ever made. It’s weird because the frontend is dressed up in minimalism and utilitarian style. The backend is powered by tightly coupled Elixir/erlang tools and python scripts. What the frontend does and what the backend is capable of, is entirely contradictory. It is both simple and complex.

I’ve made a lot of sites, and most of them are PHP based. Wordpress or Magento are the primary suspects. These sites are great to set up, and easy work with up to a certain point. Eventually the project reaches some amount of complexity that makes it impossible to work with a CMS. The frontend reaches max complexity, and the backend isn’t capable of handling the overload.

I’ve also made quite a few API endpoints and ReactJS or VueJS single page applications. A nodejs server, an index page, and JSON endpoints are all you need to head off to the races. The custom solutions are great, but take a longer time to set up, and often have hurdles that have already been solved. Security, abstraction, and quality assurance have all seen their share of development cycles.

Is there a best way?

Not really. It depends on who the website is for, and what developers are comfortable building on. But this one is for me, and I’m the toughest customer that I have. It’s impossible to build the perfect site, and my target for perfect is always shifting. So what’s the best way for me?

Personal Goals

  1. I wanted a platform where I could build custom content analysis tools
  2. I wanted a space where I could write my own content easily and with minimal fuss
  3. I wanted to keep it simple but powerful
  4. I wanted to make these tools accessible to as many people as possible

The alpha version

I started the process by building my first python tool. It’s a crawler and content extractor that uses the TextRank algorithm to pull out keywords. Then I put that on an endpoint with Python’s flask. It worked pretty well, but python processes were heavy, and the monolith that I was building would fail with too many requests. It would take down the server workers, database connection, and entire site.

Version 0.0.2

I continued to use Flask REST endpoints to power my python microservice, however I looked elsewhere for a frontend solution. I looked toward Javascript land. NodeJS makes development extremely easy. Javascript backend and a javascript frontend is as seamless as it gets, right? Now the database is abstracted away from the job service.

Again, the python would get slammed and die. The frontend server would have no idea that the python microservice was dead. It would lose all requests and hang frontend users.

Version 0.0.3

4 words. Redis overflow job scheduler. Npm provides a great package called Bull. It’s a Redis backed job scheduler that maintains job queues and makes sure they all get run once. Even if I destroyed my server with requests, it would find its way back to finishing the job. All good on that front.

Version 0.0.4

I wasn’t too happy that the REST API would not return job status. My job queue allowed me to store job status, but the HTTP style request and response only allowed for 2 states: started and completed. I already know when I started it and I’ve already got my data when it’s stopped. So, boo!

That’s why I implemented zeroMQ. A lightweight implementation for message passing. Now I’m storing messages, to store jobs, to run some python, then pass those messages back. With a little javascript, I could get job queue updates and pass information to the user. Instant feedback power! Now this was an implementation that worked.

Version 0.0.?

Wait, it was working! Why a new version??

Well, I tested deployment. It was a pain in my Tuchus. I had so many servers. Python server, redis server, message broker, frontend server. So many microservices that needed deployment and servers. I didn’t see a future managing all of that middleware.

Brainstorming a new solution

I went back to the drawing board and reviewed my needs. I know I wanted something that I could develop like the microservice architecture, but deploy like a monolith. Sure, there are tools that handle deployment, but I’m a solo-developer. I need both simplicity and complexity. I wanted:

  • A way to queue microservices
  • A library for interop with python
  • Overflow handling
  • Multiprocess implementation
  • Key value memory store
  • Streaming data between frontend, backend, and microservices
  • Messaging for job status
  • A simple way to deploy all that without a headache

Version 1.0.0

Here we are. Version 1. I found the solution to my problems with Elixir.

  • Actor model
  • Ports for running python scripts and passing messages
  • Mnesia (erlang version of redis) based queue
  • Supervisors for restarting processes and soft fault tolerance
  • Multicore use and messaging across any number of nodes
  • Websocket channels and rooms for server to frontend messages passing
  • Mix build tool used for scaffolding projects, reading docs, creating docs, compiling, running tests, code formatting, environment handling, profiling, dependency handling — and it’s extensible.

The site is simple, manageable, but also future proof. I can build python tools and distribute them without worrying about server overload, deployment troubles, or scaling problems.

The adult solution

The mature solution would have been an Apache stack. Spark, Mesos, Akka, Cassandra, and Kafka would have provided everything I was looking for. However, I don’t like Java very much and I think that Erlang has a very good handle on the distributed actor model.

The New Beta

This site has completely changed how I think about the web. I no longer think in terms of REST and Ajax. It’s about messages, ports, and websockets. It’s not like this is new and revolutionary. That’s how early networking was designed and built. I wish I had been working with this implementation for my ideas and projects all along.

Moving forward, this blog will develop with new ideas, tools, and features. No part of building, compiling, or writing is painful. It’s a weird feeling being so happy with my own website. Guys, I think I made a weird one, but a good one.