Node.js

"Node.js" (written and spoken guidelines)

Written and spkoen guidelines

Libraries

Unorganized:

Configuration:

Parse DOM (HTML, XML, SVG) and CSSOM:

Security:

Network:

Relative path

path.join(__dirname, "views");
path.resolve(".")// > /path/to/dir

Is parent path

Aka is subdir

const path = require('path');

function isParentPath(parent, filepath){
	const relative = path.relative(parent, filepath);
	return !!relative && relative.split(path.sep, 1)[0] !== ".." && !path.isAbsolute(relative);
}

List files recursively

import path from "path";
import {lstat, readdir} from "fs/promise";

// Walk directories
async function* getFiles(entry){
	if(!(await lstat(entry)).isDirectory()){
		return entry;
	}

	for(const subEntry of await readdir(entry)){
		yield* getFiles(path.join(entry, subEntry));
	}
};

for(const file of await getFiles("/some/path")){
	console.log(file);
}

Keep local copy of dependencies

Usefull to keep it on version control system or in context of security

  • https://github.com/JamieMason/shrinkpack

  • https://docs.npmjs.com/cli/shrinkwrap

Hard links to same dependencies across projects

In all node_modules directory on an hard drive.

  • https://github.com/jeffbski/pkglink

Turn off Spotflight indexation of dependencies

find /path/to/projects -type d  -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec xattr -w com.apple.metadata:com_apple_backup_excludeItem com.apple.backupd '{}' \;
# find /path/to/projects -type d  -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \;

Performances

Load balancer and reverse proxy

process.env.IN_PASSENGER === "1" typeof PhusionPassenger !== "undefined"

Debug

Send log infos client side

HTTP header X-ChromeLogger-Data: jsonbase64value

Profile

CPU profile file:

  • use the node --cpu-prof option that will generate a cpuprofile file

  • use Chrome Dev Tools Performance tab to read the file as flamegraph where the x-axis represents when a call happened. It annotate the source code with the sampled traces this gives an approximate time how much each line took to execute

  • use speedscope (GitHub - jlfwong/speedscope: 🔬 A fast, interactive web-based viewer for performance profiles.) to read the file as flamegraph but it merge similar call-stacks together to see where the time is being spent, the x-axis represents time consumed of the total time. It referred to as a "left-heavy" visualization. It's not a standard flamegraph where the x-axis represents when a call happened.

  • use node --cpu-prof $(which npm) run myscript to do it when run a script

Heap profile file:

Performance

Child processes

Cluster is child_process in a more convenient way to listen the same port by all children (it use child_process internally): Cluster | Node.j v9.6.1 Documentation

Standart input / output

const data = require("fs").readFileSync(0, {encoding: "utf-8"});// file 0 is stdin

Command line

Aka CLI

Shebang:

#!/usr/bin/env node

Syntax check without executing:

# https://nodejs.org/api/cli.html#cli_c_check
node --check file.js

Parse command line arguments:

Express

Test server capacity with mcollina/autocannon: fast HTTP/1.1 benchmarking tool written in Node.js

Order route registration from most specific to less specific: /foo/bar, /foo, /

import express from "express";

const app = express();
const port = process.env.PORT || 3000;

app.get('/posts/:id', async (req, res, next) => {
	try {
		const id = req.params.id;
		const value = await getPostById(id);
		res.render("posts", {id, value});
	}
	catch (error) {
		next(error);
	}
});

app.listen(port);

Internal redirect, aka reroute: node.js - Forward request to alternate request handler instead of redirect - Stack Overflow. See Express 4.x - API Reference

Reroute:

app.get("someroute", (req, res, next) => {
	// Reroute to index.html (could be handled by statics)
	req.url = "index.html";
	next("route");
});

Global error handling

At app level or a parent route level, register the error handler (a middleware with 4 arguments) after all other used middleware and routes handlers

Examples:

Express libraries

RESTful routing (resource, facet):

Get client IP with Express

req.ip

NPM and packages

Updates:

npm outdated --parseable | cut -d ':' -f 4 | xargs npm i
# or
npx npm-check-updates -u && npm i
# see also
npm-check -u
# and
npm update --latest

Security and trust

Format and lint package JSON

Prettier config to override default / projet config for package JSONs:

{
	"overrides": [
		{
			"files": "**/(package|package-lock).json",
			"options": {
				"tabWidth": 2,
				"useTabs": false,
				"endOfLine": "lf"
			}
		}
	]
}

Inspect package

# Extract package from cache
npm pack somepackagename | tail -n 1 | xargs tar -zxzf
cat package/package.json
# View registry infos
npm view somepackagename

Package variables

Aka environment variables

{
	"name": "test",
	"main": "server/index.js",
	"scripts": {
		"start": "node $npm_package_main $pm_package_config_port",
		"start-dev": "NODE_ENV=dev node $npm_package_main $pm_package_config_port"
	},
	"engines": {
		"node": ">=9.0"
	},
	"config": {
		"port": "8080"
	}
}
const mainEntry = process.env.npm_package_main;
const port = process.env.pm_package_config_port;

Install for continuous integration

npm ci

Packages version

npm outdated to list "Current", "Wanted" (aka fuzzy, latest minor release) and "Latest" (latest major release) version, use npm install <package name>@latest to install latest version

Scopes packages

Aka monorepos and multi packages

@babel/*, @angular/*, etc.

Local packages

{
  "name": "baz",
  "dependencies": {
    "bar": "file:./foo/bar"
  }
}

Fix package with patch

EACCES errors when installing global package

First: Just don't install packages globally

npm install -g packagename

Install global with node-gyp can output errors:

gyp WARN EACCES user "root" does not have permission to access the dev dir "/Users/username/.node-gyp/0.0.0"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/opt/local/lib/node_modules/packagename/.node-gyp"
gyp WARN EACCES user "root" does not have permission to access the dev dir "/opt/local/lib/node_modules/packagename/.node-gyp/0.0.0"

Use --unsafe-perm paramter for npm to fix the issue, or update the NODE_PATH env var.

See Warning "root" does not have permission to access the dev dir · Issue #454 · nodejs/node-gyp

NPM install resolved packages from HTTPS to HTTP

npm config get registry should return https://registry.npmjs.org/

  1. rm -rf node_modules/

  2. npm cache clean --force

  3. Revert the changes in your package-lock.json file

  4. npm i

Package overrides

Inspect package

# The package doesn't need to be installed
npm view <packagename> dist.tarball

Crossplatform scripts

{
	"name": "myapp",
	"config": { "port" : "3000" },
	"scripts": {
		"start": "ver && node --harmony app.js %npm_package_config_port% || node --harmony app.js $npm_package_config_port"
	}
}

Note: ver only exist in cmd.exe (default shell used on Windows). The last part (after ||) will be executed in other shell (usally /bin/sh on POSIX)

Write scripts like npm do for bins (in node_modules/.bin/):

mycommand:

#!/bin/sh
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")

case `uname` in
    *CYGWIN*|*MINGW*|*MSYS*) basedir=`cygpath -w "$basedir"`;;
esac

if [ -x "$basedir/node" ]; then
  exec "$basedir/node"  "$basedir/../path/to/mycommand_nodescript" "$@"
else
  exec node  "$basedir/../path/to/mycommand_nodescript" "$@"
fi

mycommand.cmd:

@ECHO off
GOTO start
:find_dp0
SET dp0=%~dp0
EXIT /b
:start
SETLOCAL
CALL :find_dp0

IF EXIST "%dp0%\node.exe" (
  SET "_prog=%dp0%\node.exe"
) ELSE (
  SET "_prog=node"
  SET PATHEXT=%PATHEXT:;.JS;=;%
)

endLocal & goto #_undefined_# 2>NUL || title %COMSPEC% & "%_prog%"  "%dp0%\..\path\to\mycommand_nodescript" %*

mycommand.ps1:

#!/usr/bin/env pwsh
$basedir=Split-Path $MyInvocation.MyCommand.Definition -Parent

$exe=""
if ($PSVersionTable.PSVersion -lt "6.0" -or $IsWindows) {
  # Fix case when both the Windows and Linux builds of Node
  # are installed in the same directory
  $exe=".exe"
}
$ret=0
if (Test-Path "$basedir/node$exe") {
  # Support pipeline input
  if ($MyInvocation.ExpectingInput) {
    $input | & "$basedir/node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  } else {
    & "$basedir/node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  }
  $ret=$LASTEXITCODE
} else {
  # Support pipeline input
  if ($MyInvocation.ExpectingInput) {
    $input | & "node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  } else {
    & "node$exe"  "$basedir/../path/to/mycommand_nodescript" $args
  }
  $ret=$LASTEXITCODE
}
exit $ret

See also:

Promisify

import {promisify} from 'util';
const result;
try{
	result = await promisify(obj.method.bind(obj))("value1", "value2");
	console.log(result);
}catch(error){
	console.error(error);
}

Instead of:

obj.method("value1", "value2", (error, result) => {
	if(error){
		console.error(error);
		return;
	}

	console.log(result);
})
import {readFile} from "fs/promises";
const content = await readFile("./test.txt", "utf8");

Modules

Use URI as configuration

  • https://github.com/sidorares/node-mysql2/blob/5f0fb8f1f5035e2c0207490aa2f0b838dc82fdc2/lib/connection_config.js#L166-L196

  • https://github.com/nodemailer/nodemailer/blob/5da6c87766e258f1a5fa9b628f2d9f57c9d533ce/lib/shared/index.js#L16-L91

Global error handling

  • process.setUncaughtExceptionCaptureCallback

Node sources

Zlib will don't have extra gzip header fiels (empty values):

If deflateSetHeader is not used, the default gzip header has text false, the time set to zero, and os set to 255, with no extra, name, or comment fields.

  • https://github.com/nodejs/node/blob/master/deps/zlib/zlib.h

Zlib binding:

Read UTF-8 JSON with BOM

const fs = require('fs');
cost config = fs.readFileSync('./config.json', 'utf-8');
// JSON.parse(config); fail if not start with an allowed char (`"`, `[`, `{`);
// BOM is 0xefbbbf and is considered white-space
// It can be stripped by `.trim()`:
JSON.parse(config.trim());

Require a virtual file

const fs = require("fs");
const path = require("path");
const filename = require.resolve("chrome-devtools-frontend/front_end/sdk/CookieParser.js");
// See https://github.com/ChromeDevTools/devtools-frontend/blob/4c46d0969f10f460f2a27116f4896f20f65d0989/front_end/sdk/CookieParser.js
let content = "const SDK = module.exports = {};\n" + fs.readFileSync(filename, "utf8");
const Module = require("module");// same as module.constructor
const m = new Module(filename, module/*or module.parent*/);
m._compile(content, filename);
m.filename = filename;
m.paths = Module._nodeModulePaths(path.dirname(filename));
m.loaded = true;
module.exports = m.exports;

Warning

// Log stack trace of warning like depreciation https://nodejs.org/api/util.html#util_util_deprecate_fn_msg_code
process.on("warning", warning => console.warn(warning.stack));

Depreciation

node --trace-warnings --trace-deprecation index.js
const util = require('util');
module.exports.someDepreciatedFunction = util.deprecate(() => {
	// Do something here.
}, "someDepreciatedFunction() is deprecated. Use someOtherFunction() instead.", "DEP_SOME_FUNCTION");

Path case sensitivity on Windows

Node handle pretty well path case insensibility on Windows. But if a module use a symbolic link or a junction and the working directory doesn't match the case of that path (d:\mydir instead of D:\MyDir) Node load the same module twice, for each path case. Note: NPM use junction for local path modules.

See an example:

echo module.exports = class{get __filename(){return __filename}} > Class.js
mkdir example
echo module.exports = new (require("../Class")) > example/instance.js
:: Create a junction ./junction/* <==> ./example/*
:: Note: the junction store the case used when created ("mklink /j junction .\example" vs "mklink /j junction .\Example")
mklink /j junction .\example
::dir /AL /S .
:: Main script
echo const a = require("./junction/instance"); > index.js
echo const b = require("./example/instance"); >> index.js
echo const c = require("./class"); >> index.js
echo console.log("process.cwd() =", process.cwd()); >> index.js
echo console.log("junction/instance = a"); >> index.js
echo console.log("example/instance = b"); >> index.js
echo console.log("a instanceof c =", a instanceof c); >> index.js
echo console.log("b instanceof c =", b instanceof c); >> index.js
echo console.log("a super.__filename =", a.__filename); >> index.js
echo console.log("b super.__filename =", b.__filename); >> index.js
:: Enable node module debug (request, looking and load)
::set "NODE_DEBUG=module"
:: Use powsershell to start a process with the current working directory with lowercase as working directory
powershell "Start-Process -NoNewWindow -FilePath node.exe -ArgumentList 'index.js' -Wait -WorkingDirectory $(Get-Location).ToString().ToLower()"

:: Will log:
::
:: ```
:: process.cwd() = D:\somepath\test
:: junction/instance = a
:: example/instance = b
:: a === b = false
:: a instanceof c = false
:: b instanceof c = true
:: a super.__filename = D:\SomePath\Test\Class.js
:: b super.__filename = D:\somepath\test\Class.js
:: ```
::
:: Note the difference of path case between constructors of a and b
:: A and b should be the same object, have the same constructor from ./Class.js
:: It's because the case of the instance is not the same: != path case -> != modules
:: Compare  with:
powershell "Start-Process -NoNewWindow -FilePath node.exe -ArgumentList 'index.js' -Wait

Require specific version of NPM and node

# update your package.json to add:
# "engines": {
# 	"npm": ">=6.6.0",
# 	"node": ">=12.0.0"
# },
npm config set engine-strict false --userconfig ./.npmrc

If the version doesn't match, npm install:

npm ERR! code ENOTSUP
npm ERR! notsup Unsupported engine for <package>@<version>: wanted: {"npm":">=6.6.0","node":">=12.0.0"} (current: {"node":"<current-node-version>","npm":"<current-npm-version>"})
npm ERR! notsup Not compatible with your version of node/npm: <package>@<version>
npm ERR! notsup Required: {"npm":">=6.6.0","node":">=12.0.0"}
npm ERR! notsup Actual:   {"npm":"<current-npm-version>","node":"<current-node-version>"}

npm ERR! A complete log of this run can be found in:
npm ERR!     <log-file-path>

Note: that doesn't work for npm ci that ignore that check

Encryption with Node.js

Asynmmetric encryption usally encrypt an symectric key used to encrypt: RSA maximum bytes to encrypt, comparison to AES in terms of security? - Information Security Stack Exchange

See security cryptography - "Diffie-Hellman Key Exchange" in plain English - Information Security Stack Exchange

Node IPC

const path = isWindows ? "\\\\.\\pipe\\myprogram" : "/tmp/myprogram";// "myprogram" or a random string

FNM

On Windows for Bash, add to %USERPROFILE%\.bashrc:

# See https://github.com/Schniz/fnm/tree/master#shell-setup and https://github.com/Schniz/fnm/blob/master/docs/configuration.md
eval "$(fnm env --use-on-cd --version-file-strategy=recursive --log-level=error)"

%USERPROFILE%\.bash_profile:

if [ -s ~/.bashrc ]; then source ~/.bashrc; fi

On Windows for CMD:

Exec the registry script:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor]
"AutoRun"="@%USERPROFILE%\\autorun.cmd"

%USERPROFILE%\autorun.cmd:

@echo off
:: See https://github.com/Schniz/fnm/tree/master?tab=readme-ov-file#windows-command-prompt-aka-batch-aka-wincmd and https://github.com/Schniz/fnm/blob/master/docs/configuration.md
:: "for /f" will launch a new instance of cmd so we create a guard to prevent an infnite loop
if not defined FNM_AUTORUN_GUARD (
    set "FNM_AUTORUN_GUARD=AutorunGuard"
    for /f "tokens=*" %%z in ('fnm env --use-on-cd --version-file-strategy=recursive --log-level=error') do call %%z
)

To spawn process without using a shell (node, npm, npx), use instead something like C:\path\to\node.cmd with the following content:

@echo off
fnm use --silent-if-unchanged
node %*

Max memory size

Aka --max-old-space-size, "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory", "EXEC : FATAL error : Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory"

Increase memory to 4GB: node --max-old-space-size=4096 index.js. 1024 for 1GB, 1536 for 1.5GB, 2048 for 2GB, 3072 for 3GB, 5120 for 5GB, 6144 for 6GB, 7168 for 7GB, 8192 for 8GB, etc.

It is recommended to always explicitly set the --max-old-space-size instead of relying on default imposed by Node.js

On a machine with 2 GiB of memory, consider setting this to 1536 (1.5 GiB) to leave some memory for other uses and avoid swapping.

— Command-line API - --max-old-space-size=SIZE (in megabytes)

TypeScript

Single executable application

SABs are not meant for bundling dependencies

HTTP Proxy

Node use interally the package undici, which parse the proxy URI with URL API for which the protocol is not optional:

$ HTTPS_PROXY=example.com:8081 node -e "console.log(new URL(process.env.https_proxy ?? process.env.HTTPS_PROXY))"
URL {
  href: 'example.com:8081',
  origin: 'null',
  protocol: 'example.com:',
  username: '',
  password: '',
  host: '',
  hostname: '',
  port: '',
  pathname: '8081',
  search: '',
  searchParams: URLSearchParams {},
  hash: ''
}
$ HTTPS_PROXY=https://example.com:8081 node -e "console.log(new URL(process.env.https_proxy ?? process.env.HTTPS_PROXY))"
URL {
  href: 'https://example.com:8081/',
  origin: 'https://example.com:8081',
  protocol: 'https:',
  username: '',
  password: '',
  host: 'example.com:8081',
  hostname: 'example.com',
  port: '8081',
  pathname: '/',
  search: '',
  searchParams: URLSearchParams {},
  hash: ''
}

EnvHttpProxyAgent automatically reads the proxy configuration from the environment variables http_proxy, https_proxy, and no_proxy and sets up the proxy agents accordingly. When http_proxy and https_proxy are set, http_proxy is used for HTTP requests and https_proxy is used for HTTPS requests. If only http_proxy is set, http_proxy is used for both HTTP and HTTPS requests. If only https_proxy is set, it is only used for HTTPS requests.

no_proxy is a comma or space-separated list of hostnames that should not be proxied. The list may contain leading wildcard characters (*). If no_proxy is set, the EnvHttpProxyAgent will bypass the proxy for requests to hosts that match the list. If > no_proxy is set to "*", the EnvHttpProxyAgent will bypass the proxy for all requests.

Uppercase environment variables are also supported: HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. However, if both the lowercase and uppercase environment variables are set, the uppercase environment variables will be ignored.

Need to set full URI (not just the host and the port), eg. HTTPS_PROXY=http://example.com:8000

For some libraires / tools the procotol is optional and could use http or https as default protocol

  • https://github.com/nodejs/node/blob/2cd385ef6714b24b62edf22dd2ddd756eee9d16b/deps/undici/src/lib/dispatcher/env-http-proxy-agent.js#L35-L47

  • https://github.com/nodejs/node/blob/2cd385ef6714b24b62edf22dd2ddd756eee9d16b/deps/undici/src/docs/docs/api/EnvHttpProxyAgent.md?plain=1#L7-L11

  • https://github.com/nodejs/node/blob/5ad2ca920cdc6d99a8d673724b35115fef925e78/deps/openssl/openssl/crypto/http/http_client.c#L965C17-L965C36 -> https://github.com/nodejs/node/blob/5ad2ca920cdc6d99a8d673724b35115fef925e78/deps/openssl/openssl/crypto/http/http_lib.c#L74-L84

  • NPM use proxy-agent - npm, which use proxy-from-env use the request procol as default https://github.com/Rob--W/proxy-from-env/blob/095d1c26902f37a12e22bea1b8c6b67bf13fe8cd/index.js#L47-L50 ("default to the requested URL's scheme")

  • curl use HTTP as default protocol ("head: It's a HTTP proxy") https://github.com/curl/curl/blob/4af40b3646d3b09f68e419f7ca866ff395d1f897/lib/url.c#L4608-L4638

  • wget use HTTP as default protocol ("Just prepend "http://" to URL.") https://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?id=93c1517c4071c4288ba5a4b038e7634e4c6b5482#n1283 https://git.savannah.gnu.org/cgit/wget.git/tree/src/url.c?id=93c1517c4071c4288ba5a4b038e7634e4c6b5482#n609

  • Python's urllib use request protocol as default https://github.com/python/cpython/blob/936135bb97fe04223aa30ca6e98eac8f3ed6b349/Lib/urllib/request.py#L801-L802

See also:

PreviousJavaScriptNextLanguage parsing

Last updated

Was this helpful?