Node.js

"Node.js" (written and spoken guidelines)

Libraries

Unorganized:

Configuration:

Parse DOM (HTML, XML, SVG) and CSSOM:

Security:

Network:

Relative path

path.join(__dirname, "views");
path.resolve(".")// > /path/to/dir

Is parent path

Aka is subdir

const path = require('path');

function isParentPath(parent, filepath){
	const relative = path.relative(parent, filepath);
	return !!relative && relative.split(path.sep, 1)[0] !== ".." && !path.isAbsolute(relative);
}

List files recursively

import path from "path";
import {lstat, readdir} from "fs/promise";

// Walk directories
async function* getFiles(entry){
	if(!(await lstat(entry)).isDirectory()){
		return entry;
	}

	for(const subEntry of await readdir(entry)){
		yield* getFiles(path.join(entry, subEntry));
	}
};

for(const file of await getFiles("/some/path")){
	console.log(file);
}

Keep local copy of dependencies

Usefull to keep it on version control system or in context of security

  • https://github.com/JamieMason/shrinkpack

  • https://docs.npmjs.com/cli/shrinkwrap

Hard links to same dependencies across projects

In all node_modules directory on an hard drive.

  • https://github.com/jeffbski/pkglink

Turn off Spotflight indexation of dependencies

find /path/to/projects -type d  -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec xattr -w com.apple.metadata:com_apple_backup_excludeItem com.apple.backupd '{}' \;
# find /path/to/projects -type d  -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \;

Performances

Load balancer and reverse proxy

process.env.IN_PASSENGER === "1" typeof PhusionPassenger !== "undefined"

Debug

Send log infos client side

HTTP header X-ChromeLogger-Data: jsonbase64value

Profile

CPU profile file:

  • use the node --cpu-prof option that will generate a cpuprofile file

  • use Chrome Dev Tools Performance tab to read the file as flamegraph where the x-axis represents when a call happened. It annotate the source code with the sampled traces this gives an approximate time how much each line took to execute

  • use speedscope (GitHub - jlfwong/speedscope: 🔬 A fast, interactive web-based viewer for performance profiles.) to read the file as flamegraph but it merge similar call-stacks together to see where the time is being spent, the x-axis represents time consumed of the total time. It referred to as a "left-heavy" visualization. It's not a standard flamegraph where the x-axis represents when a call happened.

  • use node --cpu-prof $(which npm) run myscript to do it when run a script

Heap profile file:

Performance

Child processes

Cluster is child_process in a more convenient way to listen the same port by all children (it use child_process internally): Cluster | Node.j v9.6.1 Documentation

Standart input / output

const data = require("fs").readFileSync(0, {encoding: "utf-8"});// file 0 is stdin

Command line

Aka CLI

Shebang:

#!/usr/bin/env node

Syntax check without executing:

# https://nodejs.org/api/cli.html#cli_c_check
node --check file.js

Parse command line arguments:

Express

Test server capacity with mcollina/autocannon: fast HTTP/1.1 benchmarking tool written in Node.js

Order route registration from most specific to less specific: /foo/bar, /foo, /

import express from "express";

const app = express();
const port = process.env.PORT || 3000;

app.get('/posts/:id', async (req, res, next) => {
	try {
		const id = req.params.id;
		const value = await getPostById(id);
		res.render("posts", {id, value});
	}
	catch (error) {
		next(error);
	}
});

app.listen(port);

Internal redirect, aka reroute: node.js - Forward request to alternate request handler instead of redirect - Stack Overflow. See Express 4.x - API Reference

Reroute:

app.get("someroute", (req, res, next) => {
	// Reroute to index.html (could be handled by statics)
	req.url = "index.html";
	next("route");
});

Global error handling

At app level or a parent route level, register the error handler (a middleware with 4 arguments) after all other used middleware and routes handlers

Examples:

Express libraries

RESTful routing (resource, facet):

Get client IP with Express

req.ip

NPM

Updates:

npm outdated --parseable | cut -d ':' -f 4 | xargs npm i
# or
npx npm-check-updates -u && npm i
# see also
npm-check -u
# and
npm update --latest

Security and trust

Format and lint package JSON

Prettier config to override default / projet config for package JSONs:

{
	"overrides": [
		{
			"files": "**/(package|package-lock).json",
			"options": {
				"tabWidth": 2,
				"useTabs": false,
				"endOfLine": "lf"
			}
		}
	]
}

Inspect package

# Extract package from cache
npm pack somepackagename | tail -n 1 | xargs tar -zxzf
cat package/package.json
# View registry infos
npm view somepackagename

Package variables

Aka environment variables

{
	"name": "test",
	"main": "server/index.js",
	"scripts": {
		"start": "node $npm_package_main $pm_package_config_port",
		"start-dev": "NODE_ENV=dev node $npm_package_main $pm_package_config_port"
	},
	"engines": {
		"node": ">=9.0"
	},
	"config": {
		"port": "8080"
	}
}
const mainEntry = process.env.npm_package_main;
const port = process.env.pm_package_config_port;

Install for continuous integration

npm ci

Packages version

npm outdated to list "Current", "Wanted" (aka fuzzy, latest minor release) and "Latest" (latest major release) version, use npm install <package name>@latest to install latest version

Scopes packages

Aka monorepos and multi packages

@babel/*, @angular/*, etc.

Local packages

{
  "name": "baz",
  "dependencies": {
    "bar": "file:./foo/bar"
  }
}

Local package patch

Promisify

import {promisify} from 'util';
const result;
try{
	result = await promisify(obj.method.bind(obj))("value1", "value2");
	console.log(result);
}catch(error){
	console.error(error);
}

Instead of:

obj.method("value1", "value2", (error, result) => {
	if(error){
		console.error(error);
		return;
	}

	console.log(result);
})
import {readFile} from "fs/promises";
const content = await readFile("./test.txt", "utf8");

Modules

EACCES errors when installing global package

First: Just don't install packages globally

npm install -g packagename

Install global with node-gyp can output errors:

gyp WARN EACCES user "root" does not have permission to access the dev dir "/Users/username/.node-gyp/0.0.0"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/opt/local/lib/node_modules/packagename/.node-gyp"
gyp WARN EACCES user "root" does not have permission to access the dev dir "/opt/local/lib/node_modules/packagename/.node-gyp/0.0.0"

Use --unsafe-perm paramter for npm to fix the issue, or update the NODE_PATH env var.

See Warning "root" does not have permission to access the dev dir · Issue #454 · nodejs/node-gyp

Use URI as configuration

  • https://github.com/sidorares/node-mysql2/blob/5f0fb8f1f5035e2c0207490aa2f0b838dc82fdc2/lib/connection_config.js#L166-L196

  • https://github.com/nodemailer/nodemailer/blob/5da6c87766e258f1a5fa9b628f2d9f57c9d533ce/lib/shared/index.js#L16-L91

Global error handling

  • process.setUncaughtExceptionCaptureCallback

Node sources

Zlib will don't have extra gzip header fiels (empty values):

If deflateSetHeader is not used, the default gzip header has text false, the time set to zero, and os set to 255, with no extra, name, or comment fields.

  • https://github.com/nodejs/node/blob/master/deps/zlib/zlib.h

Zlib binding:

Read UTF-8 JSON with BOM

const fs = require('fs');
cost config = fs.readFileSync('./config.json', 'utf-8');
// JSON.parse(config); fail if not start with an allowed char (`"`, `[`, `{`);
// BOM is 0xefbbbf and is considered white-space
// It can be stripped by `.trim()`:
JSON.parse(config.trim());

Require a virtual file

const fs = require("fs");
const path = require("path");
const filename = require.resolve("chrome-devtools-frontend/front_end/sdk/CookieParser.js");
// See https://github.com/ChromeDevTools/devtools-frontend/blob/4c46d0969f10f460f2a27116f4896f20f65d0989/front_end/sdk/CookieParser.js
let content = "const SDK = module.exports = {};\n" + fs.readFileSync(filename, "utf8");
const Module = require("module");// same as module.constructor
const m = new Module(filename, module/*or module.parent*/);
m._compile(content, filename);
m.filename = filename;
m.paths = Module._nodeModulePaths(path.dirname(filename));
m.loaded = true;
module.exports = m.exports;

Package overrides

Warning

// Log stack trace of warning like depreciation https://nodejs.org/api/util.html#util_util_deprecate_fn_msg_code
process.on("warning", warning => console.warn(warning.stack));

Depreciation

node --trace-warnings --trace-deprecation index.js
const util = require('util');
module.exports.someDepreciatedFunction = util.deprecate(() => {
	// Do something here.
}, "someDepreciatedFunction() is deprecated. Use someOtherFunction() instead.", "DEP_SOME_FUNCTION");

Path case sensitivity on Windows

Node handle pretty well path case insensibility on Windows. But if a module use a symbolic link or a junction and the working directory doesn't match the case of that path (d:\mydir instead of D:\MyDir) Node load the same module twice, for each path case. Note: NPM use junction for local path modules.

See an example:

echo module.exports = class{get __filename(){return __filename}} > Class.js
mkdir example
echo module.exports = new (require("../Class")) > example/instance.js
:: Create a junction ./junction/* <==> ./example/*
:: Note: the junction store the case used when created ("mklink /j junction .\example" vs "mklink /j junction .\Example")
mklink /j junction .\example
::dir /AL /S .
:: Main script
echo const a = require("./junction/instance"); > index.js
echo const b = require("./example/instance"); >> index.js
echo const c = require("./class"); >> index.js
echo console.log("process.cwd() =", process.cwd()); >> index.js
echo console.log("junction/instance = a"); >> index.js
echo console.log("example/instance = b"); >> index.js
echo console.log("a instanceof c =", a instanceof c); >> index.js
echo console.log("b instanceof c =", b instanceof c); >> index.js
echo console.log("a super.__filename =", a.__filename); >> index.js
echo console.log("b super.__filename =", b.__filename); >> index.js
:: Enable node module debug (request, looking and load)
::set "NODE_DEBUG=module"
:: Use powsershell to start a process with the current working directory with lowercase as working directory
powershell "Start-Process -NoNewWindow -FilePath node.exe -ArgumentList 'index.js' -Wait -WorkingDirectory $(Get-Location).ToString().ToLower()"

:: Will log:
::
:: ```
:: process.cwd() = D:\somepath\test
:: junction/instance = a
:: example/instance = b
:: a === b = false
:: a instanceof c = false
:: b instanceof c = true
:: a super.__filename = D:\SomePath\Test\Class.js
:: b super.__filename = D:\somepath\test\Class.js
:: ```
::
:: Note the difference of path case between constructors of a and b
:: A and b should be the same object, have the same constructor from ./Class.js
:: It's because the case of the instance is not the same: != path case -> != modules
:: Compare  with:
powershell "Start-Process -NoNewWindow -FilePath node.exe -ArgumentList 'index.js' -Wait

Inspect package

# The package doesn't need to be installed
npm view <packagename> dist.tarball

Require specific version of NPM and node

# update your package.json to add:
# "engines": {
# 	"npm": ">=6.6.0",
# 	"node": ">=12.0.0"
# },
npm config set engine-strict false --userconfig ./.npmrc

If the version doesn't match, npm install:

npm ERR! code ENOTSUP
npm ERR! notsup Unsupported engine for <package>@<version>: wanted: {"npm":">=6.6.0","node":">=12.0.0"} (current: {"node":"<current-node-version>","npm":"<current-npm-version>"})
npm ERR! notsup Not compatible with your version of node/npm: <package>@<version>
npm ERR! notsup Required: {"npm":">=6.6.0","node":">=12.0.0"}
npm ERR! notsup Actual:   {"npm":"<current-npm-version>","node":"<current-node-version>"}

npm ERR! A complete log of this run can be found in:
npm ERR!     <log-file-path>

Note: that doesn't work for npm ci that ignore that check

Crossplatform scripts

{
	"name": "myapp",
	"config": { "port" : "3000" },
	"scripts": {
		"start": "ver && node --harmony app.js %npm_package_config_port% || node --harmony app.js $npm_package_config_port"
	}
}

Note: ver only exist in cmd.exe (default shell used on Windows). The last part (after ||) will be executed in other shell (usally /bin/sh on POSIX)

Write scripts like npm do for bins (in node_modules/.bin/):

mycommand:

#!/bin/sh
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")

case `uname` in
    *CYGWIN*|*MINGW*|*MSYS*) basedir=`cygpath -w "$basedir"`;;
esac

if [ -x "$basedir/node" ]; then
  exec "$basedir/node"  "$basedir/../path/to/mycommand_nodescript" "$@"
else
  exec node  "$basedir/../path/to/mycommand_nodescript" "$@"
fi

mycommand.cmd:

@ECHO off
GOTO start
:find_dp0
SET dp0=%~dp0
EXIT /b
:start
SETLOCAL
CALL :find_dp0

IF EXIST "%dp0%\node.exe" (
  SET "_prog=%dp0%\node.exe"
) ELSE (
  SET "_prog=node"
  SET PATHEXT=%PATHEXT:;.JS;=;%
)

endLocal & goto #_undefined_# 2>NUL || title %COMSPEC% & "%_prog%"  "%dp0%\..\path\to\mycommand_nodescript" %*

mycommand.ps1:

#!/usr/bin/env pwsh
$basedir=Split-Path $MyInvocation.MyCommand.Definition -Parent

$exe=""
if ($PSVersionTable.PSVersion -lt "6.0" -or $IsWindows) {
  # Fix case when both the Windows and Linux builds of Node
  # are installed in the same directory
  $exe=".exe"
}
$ret=0
if (Test-Path "$basedir/node$exe") {
  # Support pipeline input
  if ($MyInvocation.ExpectingInput) {
    $input | & "$basedir/node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  } else {
    & "$basedir/node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  }
  $ret=$LASTEXITCODE
} else {
  # Support pipeline input
  if ($MyInvocation.ExpectingInput) {
    $input | & "node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  } else {
    & "node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  }
  $ret=$LASTEXITCODE
}
exit $ret

See also:

Encryption with Node.js

Asynmmetric encryption usally encrypt an symectric key used to encrypt: RSA maximum bytes to encrypt, comparison to AES in terms of security? - Information Security Stack Exchange

See security cryptography - "Diffie-Hellman Key Exchange" in plain English - Information Security Stack Exchange

Node IPC

const path = isWindows ? "\\\\.\\pipe\\myprogram" : "/tmp/myprogram";// "myprogram" or a random string

FNM

On Windows for Bash, add to %USERPROFILE%\.bashrc:

# Use fnm https://github.com/Schniz/fnm/tree/master#shell-setup
# Set in ~/.bashrc: https://github.com/Schniz/fnm/issues/351#issuecomment-749202468
#eval "$(fnm env --use-on-cd --version-file-strategy recursive)"
# Skip firt line of fnm env because on Git bash for Windows we need cygpath: https://github.com/Schniz/fnm/issues/390#issuecomment-776240883
# And ignore --use-on-cd and reimplement it or you will get the warning https://github.com/Schniz/fnm/blob/d3841ad2929dd9c688777c1d63a6a70e5197cc2b/src/commands/use.rs#L199-L206 and also to use recursive and auto install options https://github.com/Schniz/fnm/blob/d3841ad2929dd9c688777c1d63a6a70e5197cc2b/src/shell/bash.rs#L26-L54
eval "$(fnm env --log-level=error | sed 1d)"
export PATH=$(cygpath $FNM_MULTISHELL_PATH):$PATH

__fnm_use() {
#	fnm use --silent-if-unchanged --version-file-strategy recursive --install-if-missing
	fnm use --log-level=error --version-file-strategy recursive --install-if-missing
}
__fnmcd() {
	\cd "$@" || return $?
	__fnm_use
}
alias cd=__fnmcd
__fnm_use

%USERPROFILE%\.bash_profile:

if [ -s ~/.bashrc ]; then source ~/.bashrc; fi

On Windows for CMD:

Exec the registry script:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor]
"AutoRun"="@%USERPROFILE%\\autorun.cmd"

%USERPROFILE%\autorun.cmd:

:: Windows Registry Editor Version 5.00
::
:: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor]
:: "AutoRun"="@%USERPROFILE%\\autorun.cmd"
@echo off
call "%USERPROFILE%\fnm\fnm_autorun.cmd"

%USERPROFILE%\fnm\fnm_autorun.cmd:

@echo off
call "%USERPROFILE%\fnm\fnm_env.cmd"
doskey cd=%USERPROFILE%\fnm\fnm_cd.cmd $*

%USERPROFILE%\fnm\fnm_use.cmd:

@echo off
::fnm use --silent-if-unchanged --version-file-strategy recursive --install-if-missing
fnm use --log-level=error --version-file-strategy recursive --install-if-missing

%USERPROFILE%\fnm\fnm_cd.cmd:

@echo off
cd %1
call "%USERPROFILE%\fnm\fnm_use.cmd"

%USERPROFILE%\fnm\fnm_env.cmd:

:: Use fnm https://github.com/Schniz/fnm/tree/master#shell-setup
@echo off
::FOR /f "tokens=*" %i IN ('fnm env') DO CALL %i
:: And use recursive and auto install options https://github.com/Schniz/fnm/blob/d3841ad2929dd9c688777c1d63a6a70e5197cc2b/src/shell/windows_cmd/mod.rs#L30-L44 https://github.com/Schniz/fnm/blob/d3841ad2929dd9c688777c1d63a6a70e5197cc2b/src/shell/windows_cmd/cd.cmd
:: Plus this do not use a subshell! Or this script will call itself (a loop)!
set "tmpfile=%tmp%\autorun_fnm~%RANDOM%%RANDOM%%RANDOM%%RANDOM%.tmp"
fnm env --log-level=error >%tmpfile%
for /f "tokens=*" %%i in (%tmpfile%) do call %%i
del %tmpfile%

%USERPROFILE%\fnm\fnm_node.cmd:

@echo off
call "%USERPROFILE%\fnm\fnm_env.cmd"
call "%USERPROFILE%\fnm\fnm_use.cmd"
node %*

%USERPROFILE%\fnm\fnm_npm.cmd:

@echo off
call "%USERPROFILE%\fnm\fnm_env.cmd"
call "%USERPROFILE%\fnm\fnm_use.cmd"
npm %*

%USERPROFILE%\fnm\fnm_npx.cmd:

@echo off
call "%USERPROFILE%\fnm\fnm_env.cmd"
call "%USERPROFILE%\fnm\fnm_use.cmd"
npx %*

Max memory size

Aka --max-old-space-size, "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory", "EXEC : FATAL error : Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory"

Increase memory to 4GB: node --max-old-space-size=4096 index.js. 1024 for 1GB, 1536 for 1.5GB, 2048 for 2GB, 3072 for 3GB, 5120 for 5GB, 6144 for 6GB, 7168 for 7GB, 8192 for 8GB, etc.

It is recommended to always explicitly set the --max-old-space-size instead of relying on default imposed by Node.js

On a machine with 2 GiB of memory, consider setting this to 1536 (1.5 GiB) to leave some memory for other uses and avoid swapping.

— Command-line API - --max-old-space-size=SIZE (in megabytes)

TypeScript

NPM install resolved packages from HTTPS to HTTP

npm config get registry should return https://registry.npmjs.org/

  1. rm -rf node_modules/

  2. npm cache clean --force

  3. Revert the changes in your package-lock.json file

  4. npm i

Single executable application

SABs are not meant for bundling dependencies

PreviousJavaScriptNextDevelopment

Last updated