Star Wars Explorer

An exercise project to fetch, transform and explore data from SWAPI

Documentation Status CodeCov coverage Maintainability

Features

This is an exercise project that shows how I am building software.

This is a Django application backed by a core business package for data processing fetched from SWAPI. There is a clear separation between the web application and the business package.

Separation of core domain from the UI

The web application does not have any idea about core package internals, for eg. it does not know that pelt is used for data processing. The core business package does not know or depend on web application or Django.

You could build a CLI interface to perform the same operation as the web application and use the same public API of the core package. You could an GRPC interface and host the package on another server.

The storage is filesystem based but in real life should be based on some kind of file/blob storage. The storage backend should be configurable and injected into the core package by the dependency injection container.

Tests

The core package is well tested and does not rely on the SWAPI for the most part. There is a couple of integration test that does depend on the SWAPI and will fail if the SWAPI is not available.

I regret that django application is not as well tested, but good tests for web applications are time-consuming. Getting sensible coverage of all possible corner cases require hardening the interface, input data validation and mindful UX design.

Conscious overkill

In some aspects this project is an overkill. For a quick prototype, we could go with KISS and put domain functionality within the web app. We could also go with the simplest possible setup of django app. But I wanted to show at least some tricks that I learned over the years. I did not work with Django for over 2 years, so I may have used some outdated patterns.

Still a proof-of-concept

I have left a bit of a mess in the web application. In real life I would put more effort in better template organization, static assets generation. Generally I try to prepare for a long haul by finding a good balance between KISS, DRY, robustness, observability and reliability.

Missing workers and asynchronous processing

This solution is fetching and processing data from SWAPI during the request processing. This is incorrect for multiple reasons.

The correct solution would be to schedule the fetching & data processing job, publish a message to trigger a worker execution and return a job id to the web application.

Ideally the worker component would not relly on django and has its own persistence and management layer for monitoring and orchestration of jobs. Django web application should then ask this component for the status of the job.

We could go with a solution like this or pick up a tool readily available for this kind of job scheduling.

Background processing sequence diagram

Demo

The demo is running here: https://sw-explorer.herokuapp.com

You can deploy this project to Heroku yourself:

Deploy Django Opt-out example project to Heroku

Quickstart

  1. Fork the star-wars-explorer repo on bitbucket.org

  2. Clone your fork locally:

    $ git clone git@github.com:wooyek/star-wars-explorer.git

  3. Setup your development env the usual way or with pipx and pew:

    $ pipx install pew
$ cd star-wars-explorer
$ pew new -p python3 -a $(pwd) $(pwd | xargs basename)
$ pew workon star-wars-explorer
$ pip install -e .[development]

  4. Test project health:

    $ python manage.py check
$ pytest
$ make coverage

  5. Create a branch for local development and start development server:

    $ git checkout -b name-of-your-bugfix-or-feature
$ echo "127.0.0.1 dev.example.com" | sudo tee -a /etc/hosts
$ python manage.py migrate
$ python manage.py runserver dev.example.com:8000

Credits

This package was created with Cookiecutter and the wooyek/cookiecutter-django-app project template.