master commit release

About

TL;DR ExpressJS, VueJS cookbook, with evergreen recipes and templates (CRUD, CI/CD, QA, Testing, Cloud container deployment, Web Components, ES Modules, etc.) to develop applications faster, while reducing the need for rewrite or refactoring due to changes in dependencies.

Latest Version 0.6.11 - Released 2022 Jul 22 0830 +8GMT

NOTE:

NestJS, ReactJS and Typescript example projects react-admin and nest-admin have been moved to https://github.com/ais-one/cookbook-ts
solid project (SolidJS frontend) has been removed
cypress E2E test replaced with playwright

Ask for help and recommend improvements here

Considerations for this project are similar to favv. The difference between them are:

this repo is more of a cookbook and recipes are constantly being improved and updated
favv backend is written in Python, they can be used to improve on each other

Features

Folder

Description

Features

@es-labs/esm

resuable ES module codes

Available as npm package also

@es-labs/node

reusable common JS codes

Available as npm package also

docker-devenv

Docker containers
supporting local development

Mongodb, Mysql, Keycloak(SAML/OIDC, etc IDP), Kafka, Hashicorp Vault, Redis

docs

Documentation
To constantly improved/updated

- Main documentation
- Secrets documentation
- Deployment notes
- Kafka docs and code
- TCP Server docs and code

js-node/expressjs

Base ExpressJS application
(REST API, Websocket, etc)

- CORS, proxy middleware, helmet, error handling, logging, OpenAPI
- Objection ORM (removed), Knex, MongoDb, Relational DB data example, migration, seed, GraphQL, Redis
- Webpush & FCM push notification, Sendgrid email, Nexmo SMS, Telegram
- AgendaJS message queue
- Unit Test & Integration Test

js-node/expressjs/apps/app-template

Custom application (app-template)
built on Base ExpressJS application

- [config] app configs
- [controllers]
- [models]
- [openapi] OpenAPI yamls
- [routes] API routes (also websocket handlers)
- [services] services to startup/shutdown
- [tables] config tables for generic table crud (t4t)
- [tests] folder for tests
- [graphql-schema.js] application GraphQL codes

js-node/expressjs/public/demo-express

Frontend to test backend features

- GraphQL, File uploads, Signed URL file upload to GCP Storage, websockets, SSE, webworkers (frontend demo)
- JWT using RSA, JWT refresh token, token in HttpOnly cookies, GA OTP, role, Passport SAML, OIDC
- Github OAuth2 login (setup - https://www.sohamkamani.com/blog/javascript/2018-06-24-oauth-with-node-js)
- Fido & Webauthn

js-node/*

Other Backend applications

- Services (TCP event/stream, scalable Websocket with Redis Pubsub, etc.)
- Serial server
- openapi-file-joiner combine OpenAPI files

js-web/vue-nobundler

Vue 3 SPA no bundler + Bulma

- signed uploads, recaptcha
- Web component table, form & CRUD backend (files to note)

js-node/expressjs/apps/app-template/tables/	sample custom app table configurations
js-node/expressjs/router/t4t.js	handle backend CRUD API
@es-labs/esm/t4t-fe.js	frontend operations to interact with t4t.js
@es-labs/esm/t4t-validate.js	validation used by both front and backend
@es-labs/esm/bwc-table	used to display table
@es-labs/esm/bwc-t4t-form.js	form generated from table configurations
js-web/vue-nobundler/views/ui1.js	autcomplete, combobox, file upload example
js-web/vue-nobundler/views/ui2.js	table example
js-web/vue-nobundler/views/ui3.js	form example (with connection to backend)
js-web/vue-nobundler/views/ui4.js	table and form example (with connection to backend)

js-web/vue-vite

Vue 3 SPA using vite + Ant Design

- Leaflet Map, AntV Charts, PWA, Websockets, rxJS
- JWT refresh token, 2FA GA OTP, OIDC, SAML, Github OAuth
- Web Components (Webcam, Signature)
- Cypress Testing

example-webpack
(Deprecated & removed)
last updated version

Vue 2 SPA using webpack + Vuetify

- Graphql, REST, VueCrudX, i18n, rxJS

.github/workflows

Github Actions (CI/CD)

- Manually Trigger
- On-push Trigger (WIP)

AMP Website

removed

QUICK START - ON YOUR LOCAL MACHINE

Requirements

Node 16+ LTS
Npm 8.3.2+ (using workspaces)
For Windows, integrate bash shell to cmd shell (when installing git), or use git-bash
Docker

Updating npm on Windows

npm i -g npm@latest

Optional VS Code Plugins

NOTE Useful plugins if using VS Code:

Essentials
- Docker
- Live Server
- REST Client
- SSH FS
- MongoDB Client (official)
Recommended
- SonarLint (requires java)
- GitLens
JS Language Specific
- es6-string-html
- ESLint
- Volar (for VueJS)
- Prettier (disabled)

Chrome Extensions

Other Utilities

DB clients
- dbeaver (mac / windows)
- heidisql (windows)

Download

# clone repo and install backend
git clone https://github.com/ais-one/cookbook.git
cd cookbook

Installing & Updating Dependencies

Install dependencies for all workspaces!

Note

when doing npm i, it will always install latest version matching your package
sometimes you need to rebuild, delete all node_modules folders and the package-lock.json file in the root

# https://github.com/npm/cli/issues/708
# https://github.com/npm/cli/issues/2032
npm i # use this
# npm i --legacy-peer-deps # use this if there is peer dependencies issues, but not recommended

Update dependencies for all workspaces!

npm outdated # use this to check for outdated dependencies
npm update --save
npm ls <?package> # use npm ls to check on actual versions installed

Install for single workspace

npm i @vscode/sqlite3 --workspace=@es-labs/node
npm i lorem-ipsum --workspace=@es-labs/node

# updating from 2.2.8 to 3.1.1 - major version change
npm i ant-design-vue@latest --workspace=js-web/vue-vite

ExpressJS Backend Setup & Run - development environment

npm run app --workspace=js-node/expressjs -- development

NOTES

MongoDB examples needs MongoDB to work. To resolve, chose one of the methods to install MongoDB in docs/backend/mongodb/install.md
The js-node/expressjs/apps/app-template/config/secret/*.env,js files are not present. So there maybe some console log errors (but it is ok to ignore) and websockets will not work. Quick start is still usable. Use the README.md to fill up

Run migration & app

# create and seed relational db on SQLite, (delete the dev.sqlite file each time before you run this)
# command: npm run knex -- <development / uat / production> <custom app name> <seed / migrate>
npm run knex --workspace=js-node/expressjs -- development app-template migrate
npm run knex --workspace=js-node/expressjs -- development app-template seed

# create and seed MongoDB requires MongoDB - you can skip this but MongoDB examples will not work
# command: npm run mongo -- <development / uat / production> <custom app name> <seed / update>
npm run mongo --workspace=js-node/expressjs -- development app-template seed

# run the backend
# command: npm run app --workspace=js-node/expressjs -- <development / uat / production> <custom app name, default = app-template>
npm run app --workspace=js-node/expressjs -- development # app name implied (implied as app-template if not in env)
npm run app --workspace=js-node/expressjs -- development app-template # or app name specified
npm run app:lint --workspace=js-node/expressjs -- development app-template # to include eslint checks

Visit the following URLs

http://127.0.0.1:3000/api/healthcheck - app is running normally
http://127.0.0.1:3000 - Website served by Express with functional samples and demos
http://127.0.0.1:3000/api-docs - OpenAPI documentation

Note: to generate api docs, visit js-node/openapi-file-joiner and follow readme file, also look at the config properties OPENAPI_PATH and OPENAPI_VALIDATOR in js-node/expressjs/apps/app-template/config/common.env.js.

No bundler frontend

See js-web/vue-nobundler. Served from http://127.0.0.1:3000/native/index.html

Testing

To run unit & integration test on /api/categories endpoint. E2E testing is Work In Progress
TO TEST EVERYTHING PLEASE change describe.only(...) to describe(...) in the test scripts in js-node/expressjs/apps/app-template/tests

# run in development only
npm run test --workspace=js-node/expressjs -- development app-template

Long Running Processes

For long running processes such as tcp server (event mode, streaming mode), serial server, kafka producer, consumer, cron-triggered process, etc.

See js-node/README.md

Vite SPA Setup & Run - development environment

See js-web/vue-vite/README.md. To setup the configuration files. End-to-end testing example using playwright is here also.
Run the following

npm run dev --workspace=js-web/vue-vite

Visit

http://127.0.0.1:8080/ to view application
http://127.0.0.1:8080/nested/index.html to view another page (vite serving multi page, each page can be an SPA)

See js-web/vue-vite/README.md for more information on the vue-vite package
E2E Testing

npx playwright install chromium
npx playwright test --browser=chromium

Why No SSR or SSG

potential slow rendering by server app, added complexity in code, rehydration errors, added complexity in server
https://github.com/nuxt/nuxt.js/issues/8102
prefer static sites and lazy loaded SPA for now

SAML, OIDC, OAuth

Refer to link below on how to try out...

Keycloak README.md
Refer also to the following files
- js-node/expressjs/router/saml.js
- js-node/expressjs/router/oidc.js
- js-node/expressjs/router/oauth.js requires setup of github account and config setup here
You can test out on the js-web/vue-vite Signin UI,
- Use the following username / password credentials
  - for simple Login...
    - just clock on Login button, credentials test / test is already prefilled
    - then click on OTP button, the OTP 111111 is already prefilled
  - for SAML and OIDC... test-user / password
    - redirect to a keycloak IDP
  - for OAuth (Github), you need to configure your github settings (not recommended for starting out)
    - redirect to github login
  - for Mock user login, just click on Mock button

Fido2

Refer to following files for SPA sample (uses fido2-lib in backend)

Configuration

The js-node/expressjs/apps/app-template/config folder contains the config information.

You can override the configurations using <NODE_ENV>.env.js files, e.g. development.env.js or uat.env.js in the folder

Project Strcuture

+- .github/ : github related CICD and automations
+- .husky : git hooks
+- @es-labs/
|  +- esm/ : [shared es modules]
|     +- package.json
|  +- node/ : [shared cjs modules]
|     +- auth/ : authentication
|     +- comms/ : messaging
|     +- services/ : db, mq, etc.
|     +- config.default.js: defaults
|     +- config.js: config loader
|     +- package.json
|     +- traps.js
+- docker-devenv/ : docker for development environment
+- docs/ : documentation
+- js-node/ : nodejs applications (kafka, cron triggered, long running)
|  +- expressjs/ : express backend - See [js-node/expressjs/README.md](js-node/expressjs/README.md) for project structure
|  +- openapi-file-joiner/ : pre-process utility to combine openapi yaml files for use in openapi related packages
|  +- serialserver/
|  +- services/: TCP server, kafka, long running / cron processes
|  +- worker-threads/ : demo on use of worker threads
+- js-web
|  +- vue-nobundler/ : frontend (Vue3 no bundler) - See [js-web/vue-nobundler/README.md](js-web/vue-nobundler/README.md) for Project Structure
|  +- vue-vite/: frontend (Vue3 rollup) - See [js-web/vue-vite/README.md](js-web/vue-nobundler/README.md) for Project Structure
+- .editorconfig
+- .gitignore
+- BACKLOG.md
+- CHANGELOG.md
+- LICENCE
+- package.json
+- README.md

CI/CD & Cloud Deployment

Cloud Services

The following Google Cloud Platform (GCP) services are used

Container Registry
Cloud Run
Cloud Storage
Mongo Atlas (hosted on GCP)

Refer to doc/deployment/home.md for documentation on deployments

Deployment Using Github Actions

.github/workflows/manual-gcp-expressjs.yml (Manually deploy js-node/expressjs to GCP CloudRun)
- selectable inputs
  - environment (uat for now, development does not deploy anything)
  - service (default = app-template)
  - branch
.github/workflows/manual-gcp-vue-vite.yml (Manually deploy js-web/vue-vite to GCP Cloud Storage)
- selectable inputs
  - environment (uat for now, development does not deploy anything)
  - csutom_app (to be implemented)
  - branch
.github/workflows/manual-gh-pages.yml (Manually deploy js-web/vue-vite to Github Pages)
- selectable inputs
  - environment (uat for now, development does not deploy anything)
  - csutom_app (to be implemented)
  - branch

NOTE config/secret contents will not be in repo for CI/CD (so you can get errors), those should be put in VAULT

Current secrets

GCP_PROJECT_ID
GCP_SA_KEY
VAULT_uat, passed in as VAULT

# do not merge
VAULT="unused"

# connect to a hashicorp vault and obtain secrets to merge
VAULT={ url, token } # base64 encoded

 # pass in secrets, this way is insecure and not a good way to send in secrets
VAULT={ secrets: { ... all your secrets here } } # base64 encoded

VERSION CHANGE NOTES

v0.6+ Improve organization, graceful exit, logging, project rename, add more nodejs applications, repo name vue-crud-x changed to cookbook
v0.5+ Improve organization and authentication, add new features
v0.4+ Improve folders and structure organization, handle CI/CD better
v0.3+ Reorganize folders and structure, for ease of developing and maintaining multiple applications.
v0.2+ uses Vuetify 2. Due to breaking changes from Vuetify 1 to 2, CRUD component code was refactored to be more UI framework agnostic (reduce dependencies!), easier to use, improving code quality, documentation and supprting article - VueJS+ExpressJS CRUD & Cookbook
v0.1 and Vuetify 1 will be supported under the v1 branch. You can refer to the v1 Legacy Article (For Historical Reference)

UI Components > Table

cookbook 6.76

GitHub

NPM

NPMs

About

Features

QUICK START - ON YOUR LOCAL MACHINE

Requirements

Optional VS Code Plugins

Chrome Extensions

Other Utilities

Download

Installing & Updating Dependencies

Install for single workspace

ExpressJS Backend Setup & Run - development environment

Run migration & app

No bundler frontend

Testing

Long Running Processes

Vite SPA Setup & Run - development environment

Why No SSR or SSG

SAML, OIDC, OAuth

Fido2

Configuration

Project Strcuture

CI/CD & Cloud Deployment

Cloud Services

Deployment Using Github Actions

VERSION CHANGE NOTES