holi-okuna

The Okuna-based backend for HOLI.

Table of contents

• Table of contents
• Introduction
• Prerequisites
  • Geospatial libraries
• Quickstart
  • Reset
  • Step by step
• Changes to reduce dependencies
  • spectra==0.0.11
  • colormath==3.0.0
• Known installation issues
  • MacOs
• CI/CD
• Access local db
• Admin Dashboard
• Pycharm
• Services using email
• Translations
• Tests

Introduction

holi-okuna is based on / forked off okuna-api, the backend of the Okuna project. See README_OKUNA.md for the original (more extensive) README. This README here is thought for a quickstart and gives additional information about specifics of the HOLI project.

Prerequisites

The project uses

• Python 3.10+
• Django 4
• PostgreSQL (with PostGIS extension)
• Redis
• docker / docker-compose
• pyenv (optional)
• Black code formatter (see the docs for IDE integrations)

Geospatial libraries

The following geospatial libraries should be installed prior to building PostGIS.

• GEOS
• PROJ
• GDAL

You should also update your local .env and .docker-compose.env to include the geospatial libraries paths in your local machine.

GDAL_LIBRARY_PATH = "/path/to/libgdal.dylib"
GEOS_LIBRARY_PATH = "/path/to/libgeos_c.dylib"
PROJ_LIBRARY_PATH = "/path/to/libproj.dylib"

On Mac you can use

brew install gdal proj geos

and

GDAL_LIBRARY_PATH = "/opt/homebrew/lib/libgdal.dylib"
GEOS_LIBRARY_PATH = "/opt/homebrew/lib/libgeos_c.dylib"
PROJ_LIBRARY_PATH = "/opt/homebrew/lib/libproj.dylib"

Quickstart

If you have installed pyenv, you can execute ./reset_environment.sh to install the required python interpreter, set up the virtual environment and install the required dependencies.

Start the required services using python3 okuna-cli.py up-services-only

Start the dev server using python3 start_server.py --reload

Reset

Use python3 okuna-cli.py clean to reset all local state, including database and config files.

Step by step

1. Make sure you're actually running the correct Python version (see .python-version file) by running python3 --version, if you are using pyenv, the version check will be executed implicitly for you.

   Use pyenv install $PYENV_VERSION to install the currently used interpreter and delete your .venv folder to remove the deprecated environment. reset_environment.sh does what you need.

2. Install AND set up direnv and run direnv allow . in the current directory in order for direnv to initialise the development environment (variables). This should automatically create a Python 3 venv, load the environment and set some environment variables (see .envrc for details).

3. Run

   pip install -r requirements.txt
   pip install -r requirements-cli-only.txt
   python3 okuna-cli.py up-services-only

   This starts up docker-compose with the following processes:

   • scheduler
   • worker
   • db
   • redis

   
   You now need to fill some values in the created .env file. In particular, you need to fill all values that are left empty.
   If you are on an ARM Mac, you should also uncomment the POSTGRES_IMAGE entry.
   
   Then, run all the database migrations

   python3 manage.py migrate

   And fill the DB with some fixture data:

   python3 manage.py loaddata emoji-groups.json emojis.json badges.json categories.json languages.json topics.json skills.json sdgs.json moderation_categories.json feed_post_categories.json

   In addition, there is also a users.json you can load, if you want.

   You can generate more test data with a custom command:

   python3 manage.py generate_dummy_data --users 10 --spaces 5 --space_updates 3 --feed_posts 2 --insights 5

   Be aware, you need users to create the other entities with. The script uses them randomly.

   Afterwards, you're ready to start the backend API service:

   python3 start_server.py # this binds to http://127.0.0.1:8000
   
   #should you need more control you can also use:
   uvicorn openbook.asgi:application --reload [...more uvicorn options, like --host or --port...]

   Alternatively you can also start the Django development server, but this is discouraged and then file uploads don't work:

   python3 manage.py runserver localhost:8000

   (or use python3 manage.py runserver 0.0.0.0:8000 to bind to all interfaces)

Please always use python3 okuna-cli.py instead of docker-compose directly!

Why? okuna-cli.py creates a set of environment files (.okuna-cli.json, .docker-compose.env and .env). Those files contain randomly generated passwords for db and redis and the like to be used by the started processes. Also, it cleans up stuff on shutdown (docker networks & volumes, etc.), runs migrations and populates the db. If you use docker-compose directly you can easily get into inconsistent states and processes can’t talk to each other due to wrong passwords and database contents etc!

Changes to reduce dependencies

spectra==0.0.11

spectra has a dependency on numpy which takes very to load and compile. It is used in openbook_hashtags/management/commands/update_hashtags/luminance.py to change the color of a hashtag. It is also used in openbook_common/utils/helpers.py to darken a color. Since we may not use these features, the spectra import and dependency was removed.

colormath==3.0.0

colormath has a dependency on numpy and cannot be found via search in the source code.

Known installation issues

MacOs

Libmagic is not found on your mac? Try this command.

brew install libmagic

If the error still persists, libmagic might be installed under a different path, e.g. /opt/homebrew/Cellar/libmagic.

Run brew link libmagic to find out the correct path and create a symbolic link, e.g.

cd /usr/local/lib/
ln -s <correct path>/lib/libmagic.dylib libmagic.dylib

Xcode issues - Error: python@3.9: the bottle needs the Apple Command Line Tools to be installed.

 xcode-select --install

CI/CD

The CI/CD pipeline follows our normal flow, deploying branch environments and deploying main to staging automatically and to production manually. The e2e tests that should go in between staging and prod deployments are missing up until now.

Access local db

See .env for RDS_USERNAME and RDS_PASSWORD

psql -h localhost okuna -U <RDS_USERNAME>

Admin Dashboard

When using a local database, first execute python3 manage.py createsuperuser to create an admin account with your desired credentials. If you are connected to the staging database, use the credentials found in Passbolt

After starting Okuna, you can access the admin ui at http://localhost:8000/admin/

Pycharm

Shared run configurations can be found in the .run subfolder. You can use these to start your dev server or run tests.

Email and SMTP configuration

For emails we use Django's django.core.mail.backends.smtp.EmailBackend library and in order to have it work you need to set the following env variables:

EMAIL_HOST
EMAIL_HOST_PASSWORD
EMAIL_HOST_USER
DEFAULT_FROM_EMAIL
SERVER_EMAIL
SERVICE_EMAIL_ADDRESS

Translations

Django offers translation utility hooks out of the box. For us at holi, these are useful to display localized response messages to the Front-end. These "hooks" are called translation strings, and they store the extracted translation strings into a message file with the .po extension.

   # Translation Strings
_("Welcome holis")

You must generate new translations whenever you update or include a new response message using translation strings, including raise messages. This can be done in your terminal

   ./manage.py makemessages --all --no-wrap
   ./manage.py compilemessages

Tests

We are using pytest for test execution and can run the tests the following way:

pytest
# Execute a single test file
pytest openbook/tests/test_graphql.py
# Execute a single test case
pytest -k "test_correctly_updates_onboarding_steps"