The GenericSuite Scripts

GenericSuite Scripts (backend version) is a suite of features to enhance the Python API development process.

This repository contains the backend scripts necessary to build and deploy APIs made by GenericSuite (backend version) and GenericSuite AI (backend version).

Features

• AWS Deployment: Deployment to AWS as Lambda Function with API Gateway usig SAM (AWS Serverless Application Model).
• Local Development Environment: running with http or https, with or without Docker.
• Local DNS Server: to allow https API access with a domain name like app.exampleapp.local and allow access from another devices locally (e.g. smartphones) to test your App.
• Self-signed SSL certificates creation: to allow local development frontend and backend environments run over secure https connections.
• Common JSON config management: to add the Git Submodule with the common JSON config directories.
• Local MongoDB Docker conntainer: used by the test site and allows to have an offline local development environment.

Pre-requisites

• Node version 18+, installed via NVM (Node Package Manager) or NPM and Node install.
• The GenericSuite (backend version) pre-requisites.
• For AI featured APIs: The GenericSuite AI (backend version) installation guide.

Getting Started

To get started with GenericSuite, follow these steps:

Initiate your project

Check The GenericSuite Getting Started guide for more details.

Install the GenericSuite Backend Scripts

npm init

npm install -D genericsuite-be-scripts

To generate sef-signed SSL certificates, office-addin-dev-certs is required:

npm install -D office-addin-dev-certs

Prepare the Makefile

Copy the Makefile template from node_modules/genericsuite-be-scripts:

cp node_modules/genericsuite-be-scripts/Makefile ./Makefile

AWS SAM

Prepare the AWS/SAM templates

Create the project's scripts/aws_big_lambda and scripts/aws directories and copy the templates:

make init_sam

Or...

sh node_modules/genericsuite-be-scripts/scripts/aws_big_lambda/init_sam.sh

If you're going to develop with the Chalice framework:

Create the .chalice directory and copy the .chalice/config-example.json template:

make init_chalice

Or...

sh node_modules/genericsuite-be-scripts/scripts/aws/init_chalice.sh

Customize SAM Templates

If you need to do any customization to the samconfig.toml:

Edit the template-samconfig.toml file:

vi scripts/aws_big_lambda/template-samconfig.toml
# or
# code scripts/aws_big_lambda/template-samconfig.toml

Check for some customization needed.

NOTE: The deployment script big_lambdas_manager.sh will replace APP_NAME_LOWERCASE_placeholder with the Application name defined in the APP_NAME variable in .env file.

Add new Endpoints to SAM Template

When you need to add new endpoints to your App:

Edit the template-sam.yml:

vi scripts/aws_big_lambda/template-sam.yml
# or
# code scripts/aws_big_lambda/template-sam.yml

In this file is where the endpoints are defined, as well as other SAM deployment elements like the environment variables used by the AWS Lambda function. You can add new endpoints or customize it as you needed.

There's an endpoint definition template in the file node_modules/genericsuite-be-scripts/scripts/aws_big_lambda/template-sam-endpoint-entry.yml.

Be careful about elements ending with _placeholder because they are replaced by the deployment script big_lambdas_manager.sh with the corresponding values.

Add new Environment Variables

If you need to add additional environment variables to your App:

Edit the update_additional_envvars.sh file:

vi scripts/aws/update_additional_envvars.sh
# or
# code scripts/aws/update_additional_envvars.sh

Add your additional environment variables replacements in scripts/aws/update_additional_envvars.sh as:

perl -i -pe"s|ENVVAR_NAME_placeholder|${ENVVAR_NAME}|g" "${CONFIG_FILE}"

... replacing "ENVVAR_NAME" with the name of the environment variable

Add the additional environment variables to the .env file:

ENVVAR_NAME=ENVVAR_VALUE

... replacing "ENVVAR_NAME" with the name of the environment variable and ENVVAR_VALUE with its value.

Add the additional environment variables to the scripts/aws_big_lambda/template-sam.yml file, in the APIHandler > Properties > Environment > Variables section. E.g.

      .
      .
   APIHandler:
         .
         .
      Properties:
            .
            .
         Environment:
         Variables:
            ENVVAR_NAME: ENVVAR_VALUE
                  .
                  .

... replacing "ENVVAR_NAME" with the name of the environment variable and ENVVAR_VALUE with its value.

If you're using the Chalice framework, add the additional environment variables to the .chalice/config-example.json file, in the main environment_variables section. E.g.

{ "version": "2.0", . . "environment_variables": { . . "ENVVAR_NAME": "ENVVAR_NAME_placeholder" }, "stages": { . . ... replacing "ENVVAR_NAME" with the name of the environment variable (in both places).

Usage

Start Development Server

To start the development server for the dev stage and a local docker MongoDB container:

bash make run

To start the development server for the qa stage and MongoDB Atlas:

bash make run_qa

When there are changes to the dependencies or .env file, restart the local development server:

bash make restart_qa

Deploy QA

To perform a QA deployment as an AWS Lambda Function and AWS API Gateway:

make deploy_qa

To link backend API to the App domain:

• Go to Route 53.
• Click on the Zone corresponding to the App domain.
• Click on 'Create Record'.
• Enter the subdomain: 'api-qa', 'api-staging', 'api-demo' or 'api' (for production).
• Enable 'alias'.
• In 'Route traffic to' select the 'Alias to API Gateway API' option.
• In 'Choose region' select the App region.
• In 'Choose endpoint' select the one corresponding to App domain.
• Click on 'Create Records'.

Install dependencies

• Install default package categories from Pipfile.
  Runs pipenv install.
  Reference: https://pipenv.pypa.io/en/latest/commands.html#install

make install

• Install both develop and default package categories from Pipfile.
  Runs pipenv install --dev.

make install_dev

• Install from the Pipfile.lock and completely ignore Pipfile information.
  Runs pipenv install --ignore-pipfile.

make locked_install

• Install both develop and default package categories from the Pipfile.lock and completely ignore Pipfile information.
  Runs pipenv install --dev --ignore-pipfile.

make locked_dev

• Generates the requirements.txt file.
  Runs sh scripts/aws/run_aws.sh pipfile.

make requirements

or...

make lock_pip_file

• Clean install.
  Alias that runs make clean_rm and make install.

make fresh

Cleaning

• Alias to run make clean_rm, make clean_temp_dir, and make clean_logs.

make clean

• Remove a virtual environment created by "pipenv run".
  Runs pipenv --rm.

make clean_rm

• Clean logs (in /logs directory).
  Runs sh scripts/clean_logs.sh.

make clean_logs

• Clean logs, cache and temporary files.
  Runs sh scripts/aws/run_aws.sh clean.

make clean_temp_dir

CLI Utilities

• Install development tools (pyenv, pipenv, make, and optionally: poetry, saml2aws).
  Check node_modules/genericsuite-be-scripts/scripts/install_dev_tools.sh for more details about how to configure via .env file.

make install_tools

• Show ports in use.
  Runs sh scripts/run_lsof.sh.

make lsof

Automated Testing

• Start the local MongoDB docker container and run the tests.
  Runs sh scripts/run_app_tests.sh.

make test

• Execute the test without starting the local MongoDB docker container.
  Runs sh scripts/aws/run_tests.sh.

make test_only

Linting

• Execute Prospector.
  Runs pipenv run prospector.

make lint

• Execute MyPy.
  Runs pipenv run mypy ..

make types

• Execute Coverage.
  Runs pipenv run coverage run -m unittest discover tests and pipenv run coverage report.

make coverage

• Execute Yapf Formatter and PyCodeStyle.
  Runs pipenv run yapf -i *.py **/*.py **/**/*.py and pycodestyle.
  References:
  
• https://github.com/google/yapf
  
• https://pycodestyle.pycqa.org/en/latest/
  

make format

• Execute Yapf (in "print the diff for the fixed source" mode) and PyCodeStyle.
  Runs pipenv run yapf --diff *.py **/*.py **/**/*.py and pycodestyle.

make format_check

Development Commands

• Perform a complete Lint, Type check, unit and integration test, format check, and styling before deployments.
  Alias to run make lint, make types, make tests, make format_check, and make pycodestyle.

make qa

• Start the local MongoDB docker container (used for testing and dev stage run).
  Runs sh scripts/mongo/run_mongo_docker.sh run.

make mongo_docker

NOTES:
Use mongodb://root:example@mongo:27017/ as URL to connect to the local MongoDb database.
Use http://localhost:8081 to access the MongoDb Admin UI.
Use user: admin and password: pass as credentials to access the Admin UI.

• Stop the local MongoDB docker container.
  Runs sh scripts/mongo/run_mongo_docker.sh down

make mongo_docker_down

• Backup a mongoDB database.
  Runs sh scripts/mongo/db_mongo_backup.sh ${STAGE} ${BACKUP_DIR}
  

make mongo_backup

E.g.

STAGE=qa BACKUP_DIR=./dumps make mongo_backup

• Restore a mongoDB database.
  Run sh scripts/mongo/db_mongo_restore.sh ${STAGE} ${RESTORE_DIR}

make mongo_restore

E.g.

STAGE=qa RESTORE_DIR=./dumps/bkp-mongodb-[exampleapp]_[stage]-[date]_[time].zip make mongo_restore

Chalice Specific Commands

• Set parameters on .chalice/config.json as the production stage.
  Runs sh scripts/aws/set_chalice_cnf.sh prod.

make config

• Set parameters on .chalice/config.json with no specific stage.
  Runs sh scripts/aws/set_chalice_cnf.sh.

make config_dev

• Set parameters on .chalice/config.json as the Development stage.
  Runs sh scripts/aws/set_chalice_cnf.sh mongo_docker.

make config_local

• Set parameters on .chalice/config.json as the QA stage with CORS specific variables replacement, to allow use the QA live database from the local development environment.
  Runs sh scripts/aws/set_chalice_cnf.sh qa.
  References:
• APP_CORS_ORIGIN_QA_CLOUD and APP_CORS_ORIGIN_QA_LOCAL in the .env.example file.
  

make config_qa

• Set parameters on .chalice/config.json to prepare the QA deployment.
  Runs sh scripts/aws/set_chalice_cnf.sh qa deploy,

make config_qa_for_deployment

• Set parameters on .chalice/config.json as the Staging stage.
  Runs sh scripts/aws/set_chalice_cnf.sh staging.

make config_staging

• Create the AWS Stack via Chalice command.
  Runs sh scripts/aws/run_aws.sh create_stack.

make build

• Generates the requirements.txt.
  Runs sh scripts/aws/run_aws.sh pipfile.

make build_local

• Describe the AWS stack with the Chalice command.
  Runs sh scripts/aws/run_aws.sh describe_stack.

make build_check

• Alias to run make unbuild_qa.

make unbuild

• Delete the Chalice QA App.
  Runs sh scripts/aws/run_aws.sh delete_app qa.

make unbuild_qa

• Delete the Chalice Staging App.
  Runs sh scripts/aws/run_aws.sh delete_app staging

make unbuild_staging

• Delete the AWS stack created by Chalice.
  Runs sh scripts/aws/run_aws.sh delete_stack

make delete_stack

AWS S3 and other

Create the AWS S3 Buckets for different environments and other AWS utilities.

• Create S3 bucket for development.
  Runs sh scripts/aws/create_chatbot_s3_bucket.sh dev.

make create_s3_bucket_dev

• NOTES:

  • The create_s3_bucket_* scripts also allows to create or re-assign the S3 Bucket Policy.

  • If you receive the AWS_S3_CHATBOT_ATTACHMENTS_CREATION is not set to 1 message, set that environment variable or run the script in this way:

    bash AWS_S3_CHATBOT_ATTACHMENTS_CREATION=1 make create_s3_bucket_dev

• Create S3 bucket for QA.
  Runs sh scripts/aws/create_chatbot_s3_bucket.sh qa.

make create_s3_bucket_qa

• Create S3 bucket for Staging.
  Runs sh scripts/aws/create_chatbot_s3_bucket.sh staging.

make create_s3_bucket_staging

• Create the Production S3 buckets.
  Runs sh scripts/aws/create_chatbot_s3_bucket.sh prod.

make create_s3_bucket_prod

• Generate DynamoDB table definitions
  

For CF (CloudFormation) deployment:

Read all table definitions from the JSON configuration directory and generates a file that can be used as AWS Cloud Formation template.

make generate_cf_dynamodb

For SAM (Serverless Application Model) deployment:

Read all table definitions from the JSON configuration directory and generates a template file that can be inserted in the SAM template.yml file.

make generate_sam_dynamodb

• Create a default ${HOME}/.aws/config AWS configuration.
  Runs sh scripts/aws/create_aws_config.sh.

make create_aws_config

Secrets

• Generate a new seed for the storage assets URL masking

To assign the STORAGE_URL_SEED environment variable.

make generate_seed

Deployment

• Deploy the App on QA.
  Runs make create_s3_bucket_qa, sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy qa

make deploy_qa

• Validate the SAM deployment templates on QA.
  Runs make create_s3_bucket_qa, sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_validate qa

make deploy_validate_qa

• Create the deployment QA package only.
  Useful to check the package size and test the image by a local Docker run.
  Runs make create_s3_bucket_qa, sh scripts/aws_big_lambda/big_lambdas_manager.sh package qa.

make deploy_package_qa

• Deploy the App on Staging.
  Runs make create_s3_bucket_staging, sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy staging

make deploy_staging

• Deploy the App on Production.
  Runs make create_s3_bucket_prod, sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy prod

make deploy_prod

• Alias to run make deploy_qa.

make deploy

Application Specific Commands

• Execute the App locally using the development database, asking to run it over http or https.
  Runs make config_qa, make clean_logs, and sh scripts/aws/run_aws.sh run_local. [???]

make run

• Execute the App locally using the QA database, asking to run it over http or https.
  Runs make config_qa, make clean_logs, and sh scripts/aws/run_aws.sh run_local qa.

make run_qa

• Runs make config_qa, make clean_logs, and sh scripts/secure_local_server/run.sh "down" "" Stop and destroy the App local Docker container (for any running stage).
  

make down_qa

• Restart the App local Docker container running over QA.
  Runs make config_qa, make clean_logs, sh scripts/secure_local_server/run.sh "down" "" and sh scripts/aws/run_aws.sh run_local qa.

make restart_qa

• Execute the App locally using the development database (in a local Docker container), asking to execute it over http or https.
  Runs make config_local, make clean_logs, sh scripts/aws/run_aws.sh run_local dev. [???]

make run_local_docker

• Execute chalice local --port \$PORT --stage PROD
  Runs make config, make clean_logs, sh scripts/aws/run_aws.sh run.

make run_prod

Common JSON config

• Add the Git Submodule with the common JSON config directories.
  Runs sh scripts/add_github_submodules.sh.

make add_submodules

Local DNS Server

• Start the local DNS Server.
  Runs sh scripts/dns/run_local_dns.sh

make local_dns

• Runs sh scripts/dns/run_local_dns.sh restart to restart the local DNS Server.
  

make local_dns_restart

• Restart and rebuild the local DNS Server configuration when the local IP or any DNS parameters has been changed.
  Runs sh scripts/dns/run_local_dns.sh rebuild

make local_dns_rebuild

• Stop and destroy the local DNS Server.
  Runs sh scripts/dns/run_local_dns.sh down.

make local_dns_down

• Test the local DNS Server.
  Runs sh scripts/dns/run_local_dns.sh test.

make local_dns_test

Self-signed SSL certificates

• Create the self-signed local SSL certificates (required to run the local development frontend and backend over https).
  Runs sh scripts/local_ssl_certs_creation.sh.

make create_ssl_certs_only

• Copy the self-signed local SSL certificates to the frontend directory/local repository.
  Runs sh scripts/local_ssl_certs_copy.sh.

make copy_ssl_certs

• Alias to run make create_ssl_certs_only and make copy_ssl_certs.
  

make create_ssl_certs

NPM library scripts

• Update the package.json file with the version and all other parameters except dependencies.
  Runs npm install --package-lock-only.

make lock

• Test the publish to NPMJS without actually publishing.
  Runs sh scripts/npm_publish.sh pre-publish.

make pre-publish

• Publish the scripts library to NPMJS.
  Runs sh scripts/npm_publish.sh publish.
  Requirements:
  
• NpmJS Account.

make publish

Pypi library scripts

• Build 'dist' directory needed for the Pypi publish.
  Runs poetry lock --no-update, rm -rf dist and python3 -m build.
  Requirements:
  
• poetry.

make pypi-build

• Pypi Test publish.
  Runs make pypi-build, and python3 -m twine upload --repository testpypi dist/*.
  Requirements:
  
• twine.
  
• TestPypi Account.

make pypi-publish-test

• Pypi Production publish
  Runs make pypi-build, and python3 -m twine upload dist/*.
  Requirements:
  
• twine.
  
• Pypi Account.

make pypi-publish

Troubleshooting

• If you get the error Warning: Python >=3.9,<4.0 was not found on your system... doing make install:

make install

... and the response is like:

pipenv install
Warning: Python >=3.9,<4.0 was not found on your system...

You can specify specific versions of Python with:
$ pipenv --python path/to/python
make: *** [install] Error 1

Fix it with these commands:

# Set the project Python version with pyenv
pyenv local 3.11

# Set the Python path with pipenv
pipenv --python ${HOME}/.pyenv/shims/python

And repeat make install

• If you get the warning This version of npm is compatible with lockfileVersion@1... doing make install:

npm install

... and the response is like:

npm WARN read-shrinkwrap
This version of npm is compatible with lockfileVersion@1,
but package-lock.json was generated for lockfileVersion@3.
I'll try to do my best with it!

It's because you're using an old Node version. To solve it:

nvm node 18

And repeat make install

• If you get APP_NAME not set message doing any make run, it's because the .env file must be created or reviewed. Check the GenericsSuite (backend version) Configuration or GenericsSuite AI (backend version) Configuration

• If you get CORS errors in the frontend and backend communication:

• To make both use localhost and http, change these variables in the .env file:

# Frontend .env file:
APP_LOCAL_DOMAIN_NAME=localhost

# Backend .env file:
APP_CORS_ORIGIN_QA_LOCAL=http://localhost:3000

And make run both frontend and backend with the http option.

1. To make both use the local DNS server and https, change these variables in the .env file:

# Frontend .env file:
APP_LOCAL_DOMAIN_NAME=app.exampleapp.local

# Backend .env file:
APP_CORS_ORIGIN_QA_LOCAL=https://app.exampleapp.local:3000

NOTE: replace exampleapp with your App name, all lowercased.

And make run both frontend and backend with the https option.

• If the local DNS Server seems to be unreachable or not working:

Restart the local backend server:

make local_dns_restart

If the local IP changes, make sure to: 1) Run make local_dns_rebuild.

2) Copy the IP address reported by the previous command.
E.g.
Local DNS domain 'app.exampleapp.local' is pointing to IP address '192.168.1.158'.

3) Run make restart_qa

4) Add the IP address to the DNS Servers in your computer's Network > DNS servers settings. The new DNS Server IP address must be the first one in the list of DNS servers.

5) Restart the computer's WiFi or LAN network connection.

License

GenericSuite is open-sourced software licensed under the ISC license.

Credits

This project is developed and maintained by Carlos J. Ramirez. For more information or to contribute to the project, visit The GenericSuite Scripts (backend version) on GitHub.

Happy Coding!