CtrlK

Network

PreviousWeb NextTools

Last updated 2 years ago

Was this helpful?

DNS

Encrypted DNS

macOS and iOS configuration profiles:

DNS over HTTPS aka DoH:

show information in Firefox: about:networking
application/dns-message content type
RFC8484

<?php
if (!isset($_GET['token']) || $_GET['token'] != 'some-random-token')
{
	die('Invalid token');
}

// From https://github.com/NotMikeDEV/DoH
if (isset($_SERVER['CONTENT_TYPE']) && $_SERVER['CONTENT_TYPE'] == 'application/dns-message')
{
        $request = file_get_contents("php://input");
        header("Content-Type: application/dns-message");
        $s = fsockopen("udp://127.0.0.1", 53, $errno, $errstr);
        if ($s)
        {
                fwrite($s, $request);
                echo fread($s, 4096);
                fclose($s);
        }
}
else if (isset($_GET['dns']))
{
        $request = base64_decode(str_replace(array('-', '_'), array('+', '/'), $_GET['dns']));
        header("Content-Type: application/dns-message");
        $s = fsockopen("udp://127.0.0.1", 53, $errno, $errstr);
        if ($s)
        {
                fwrite($s, $request);
                echo fread($s, 4096);
                fclose($s);
        }
}
?>

DNS over TLS aka DoT:

DNS over TLS — Wikipedia

DNS records

Names (exemples for example.com):

@ naked name (example.com), the domain itself
* fallback subdomain (*.example.com, but doesn't match *.test.example.com if test.example.com is declared)
www subdomain (www.example.com)
test.www subsubdomain (test.www.example.com)
*.www fallback subsubdomain (*.www.example.com)

* CNAME @ is not possible with all DNS servers, and not recommended (need 2 DNS requests to resolve subdomain)

@ IN A 192.0.2.1
@ IN TXT "some text"

# dig www.example without DNS cache
dig @$(dig example.com NS +short | head -n1) www.example.com ANY +noall +answer

Local gTLD

*.test (purpose: "testing of current or new DNS related code")
*.localhost (but need a change in hosts file) for local only
*.dev own by Google, HSTS preload is enable (HTTPS is forced)
*.local not recommended
might be used for mDNS, mostly with Apple hardware (Bonjour).
Any DNS query for a name ending with ".local." MUST be sent to the mDNS IPv4 link-local multicast address 224.0.0.251 (or its IPv6 equivalent FF02::FB).
*.intranet
*.internal
*.private
*.corp
*.home
*.lan

HTTP

Note: don't name your local server *.local. See Local TLD Note: Range header could be ignore if the content need to be gzipped (It's the case with nginx)

FQDN: fully qualified domain name, with the trailing dot (root zone) TLD: top level domain (e.g. .com, .net, .bmw, .us). See the list of TLDs eTLD: effective top level domain (e.g. .com, .co.uk and .pvt.k12.wy.us). See the public suffix list eTLD+1: effective top level domain plus one level (e.g. example.com, example.co.uk) SLD: second level domain (e.g. co is the SLD of www.example.co.uk)

Headers

X- prefix is discouraged ("SHOULD NOT") if this header would come a standard

Use right status

401 Unauthorized: concern authentification
403 Forbidden: concern authorization
File:Http-headers-status.svg - Wikimedia Commons
https://restpatterns.mindtouch.us/HTTP_Status_Codes

Redirects can break `POST`

...or any other method other than GET.

23:50:52:658: Network: POST http://host/folder [HTTP/1.1 301 Moved Permanently 947ms]
23:50:53:614: Network: GET http://host/folder/ [HTTP/1.1 200 OK 400ms]

When you use a 301 or 302 redirect, the browser will change the method on the redirect to be a GET request, even if the original method was a POST request.

Use the 307 and 308 status codes instead if you need to retain the method

HTTPS (TLS)

See Transport Layer Security and Decrypt TLS

About mixed content:

Content-Security-Policy: upgrade-insecure-requests; - see CSP: upgrade-insecure-requests - HTTP | MDN
Troy Hunt: Disqus' mixed content problem and fixing it with a CSP
The Complete Guide To Switching From HTTP To HTTPS – Smashing Magazine
See also HSTS

Streaming and Push

Server Side Event, WebSocket or long-polling HTTP request, pull

For live content (live number of watchers, real-time transport traffic, real-time updated article, etc.), doesn't require bi-directional (full-duplex) communication, use SSE first

Note: proxy servers and firewalls could buffer the response, increasing the latency of the message delivery. Antivirus software may block the event streaming data chunks. TLS could be use to solve this problem:

Note: don't forget to push/flush the content: - php - Server sent events work, but with a massive time delay - Stack Overflow

Server Side Event

Aka SSE, EventSource

SSE use vanilla HTTP (eg thru load-balancers) and do auto-reconnect

Browser APIs and Protocols: Server-Sent Events (SSE) - High Performance Browser Networking (O'Reilly)
Stream Updates with Server-Sent Events - HTML5 Rocks
Using server-sent events - Web APIs | MDN
Server-Sent Events explained with usecases
- Example of Server-sent events with Node.js

Websocket

Aka WS

Fanout Blog » You might not need a WebSocket
WebSockets, caution required!
600k concurrent websocket connections on AWS using Node.js - blog. - 600k concurrent websocket connections on AWS using Node.js (2015) | Hacker News
websocket server library used by Stack Overflow: https://github.com/StackExchange/NetGain

For Nodejs:

Starting Node.js using forever with --nouse-idle-notification and other flags - Stack Overflow

For Ngnix:

Example of Ngnix-as-proxy configuration:

http {
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'' close;
	}

	upstream ws_server {
		server localhost:9000;
	}

	server {
		ssl on;
		ssl_certificate /path/to/crt;
		ssl_certificate_key /path/to/key;

		root /data/www/public;

		location / {
		}

		location /app {
			proxy_pass http://ws_server;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		}
	}
}

For Apache:

Example of Apache-as-proxy configuration (require mod_proxy and mod_proxy_wstunnel - require httpd 2.4.5)

# Reverse proxy to nodejs
<Location /ws-service>
	Require all granted
	ProxyPass ws://localhost:9000
	ProxyPassReverse ws://localhost:9000
</Location>

https://stackoverflow.com/questions/30443999/how-to-add-mod-proxy-wstunnel-to-apache2-2-2-on-raspberry-pi-backport-mod-proxy

Content type

Note: use text/html even if it's a fragment of a HTML document

HSTS

To disable HSTS use the following header (when HTTPS): Strict-Transport-Security: max-age=0 (in .htaccess: Header set Strict-Transport-Security "max-age=0")

See HSTS

Server Push

Aka HTTP/2 Server Push

Adviced order, push while processing (while waiting for the app response): static assets (script, image, style, fetch, etc.) then the html (let the time to the server to processing the request and render the response). Possible to send status 103 Early Hints then 200 OK Concatenate small files

Browsers often implent HTTP/2.0 only over TLS

Required Assets: CSS, JavaScript, and image assets required by a web page
Uncacheable Assets: Pushing dynamic assets = reduce page load time
Likely Next Pages: the page that a user is likely to load (e.g., the link they’re hovering over), make it look like it loads instantaneously
Redirects

<Location /index.html>
   Header add Link "</css/site.css>;rel=preload;as=style"
   Header add Link "</images/logo.jpg>;rel=preload;as=image"
</Location>

Inline Link: </css/my.css>;rel=preload;as=style, </js/jquery.js>;rel=preload;as=script works too

Or define it via PHP:

<?php
header('Link: </css/my.css>;rel=preload;as=style', false);
?>

Note: I you don't want the server push the resource (preload only) with Link header, use nopush as: Link: </css/image.webp>;rel=preload;as=image;type=image/webp;nopush Use this if the header Save-Data: on It's usefull in case the resource format is not widely supported

Cache-aware server push

<?php
// https://css-tricks.com/cache-aware-server-push/
function pushAssets() {
	$pushes = array(
		'/css/styles.css' => substr(md5_file('/var/www/css/styles.css'), 0, 8),
		'/js/scripts.js' => substr(md5_file('/var/www/js/scripts.js'), 0, 8)
	);

	if (!isset($_COOKIE['h2pushes'])) {
		$pushString = buildPushString($pushes);
		header($pushString);
		setcookie('h2pushes', json_encode($pushes), 0, 2592000, '', '.myradwebsite.com', true);
	} else {
		$serializedPushes = json_encode($pushes);

		if ($serializedPushes !== $_COOKIE['h2pushes']) {
			$oldPushes = json_decode($_COOKIE['h2pushes'], true);
			$diff = array_diff_assoc($pushes, $oldPushes);
			$pushString = buildPushString($diff);
			header($pushString);
			setcookie('h2pushes', json_encode($pushes), 0, 2592000, '', '.myradwebsite.com', true);
		}
	}
}

function buildPushString($pushes) {
	$pushString = 'Link: ';

	foreach($pushes as $asset => $version) {
		// TODO add "as" parameter (style, script, etc.)
		$pushString .= '<' . $asset . '>; rel=preload';

		if ($asset !== end($pushes)) {
			$pushString .= ',';
		}
	}

	return $pushString;
}

// Push those assets!
pushAssets();
?>

Force download

Aka generated filename

use a filename that clearly mention the website or the service that allow its identification, and the content type (invoice, contrat, etc). — Internal names for downloadable files make it possible to identify their content and origin. - Check-list The Web Quality Assurance Check-list - Opquast Check-lists

Content-Disposition: attachment; filename="file.ext"

You can use also:

Content-Type: application/octet-stream

X-Download-Options: noopen

Content-Disposition - HTTP | MDN

Access

XSS and injection prevention - CORS, CSP, SI, etc.

Referer

See Security

Method

Aka POST, GET, HEAD, PUT, etc.

An HTTP method is safe if it doesn't alter the state of the server. In other words, a method is safe if it leads to a read-only operation. Several common HTTP methods are safe: GET, HEAD, or OPTIONS.
— Safe (HTTP Methods) - MDN Web Docs Glossary: Definitions of Web-related terms | MDN

See RESTful and API

X-HTTP-Method (Microsoft), X-HTTP-Method-Override (Google/GData), X-Method-Override (IBM)
The Definitive Guide to GET vs POST - Treehouse Blog
Safe (HTTP Methods) - MDN Web Docs Glossary: Definitions of Web-related terms | MDN

Simple HTTP Server

Aka simple server

Run in controlled environments only!. Useful for testing purposes or for application demonstrations

http.server — HTTP servers — Python 3 documentation python -m http.server 8000 -d /path/to/wwwroot
PHP: Built-in web server - Manual - php -S localhost:8000 -t /path/to/wwwroot
HTTP | Node.js Documentation - const http = require("http"); http.createServer((req, res) => {res.writeHead(200, {"Content-Type": "text/html"}); res.write("Hello World!"); res.end();}).listen(8000);
- while true; do nc -l 8000 < response.http; done. Will serve an unique stream/page. To get the raw HTTP response, use printf "GET / HTTP/1.1\nHost:example.com\nUser-Agent:Firefox\Connection: close\n\n" | nc example.com 80 > response.http or write/edit it with text editor
- Written in Go
()

With Python you can create a file in server dir root MySimpleHTTPServer.py:

#!/usr/bin/python

import http.server
import socketserver

PORT = 8000

Handler = http.server.SimpleHTTPRequestHandler

with socketserver.TCPServer(("", PORT), Handler) as httpd:
	print("serving at port", PORT)
	httpd.serve_forever()

Then execute in the same folder:

python -m MySimpleHTTPServer

Content encoding

See Precompress

Note: sometimes you can't encode content, event if the final client support it, mediators could not (proxies, cache servers, etc.): Who’s not getting gzip? | High Performance Web Sites

Aka brotli, gzip, zlib, deflate

gzip is the GZIP file format: GZIP headers (can contains filename and a timestamp), DEFLATE data, and a checksum
deflate is actually the ZLIB data format: zlib header, DEFLATE data, and a checksum

Some clients do also accept the actual DEFLATE data format for deflate.

Note: GZip remove the advantage UTF-16 might have over UTF-8 for East Asian-heavy text.

About brotli vs gzip:

Brotli is mostly nice for clients, with decompression performance comparable to gzip while significantly improving the compression ratio. These are powerful properties for serving static content such as fonts and html pages. However compression performance, at least for the current implementation, is considerably slower than gzip, which makes Brotli unsuitable for on-the-fly compression in http servers or other data streams. — OpenCPU - Compression Benchmarks: brotli, gzip, xz, bz2

Cache

Aka cache policy

Sending last-modified: Thu, 01 Jan 1970 00:00:01 GMT prevents the page from updating. Gives it super long cache heuristic, and makes revalidation think it never ever changes.
— kornel@mastodon.social on Twitter: "New fun failure mode added to my collection: Sending last-modified: Thu, 01 Jan 1970 00:00:01 GMT prevents the page from updating. Gives it super long cache heuristic, and makes revalidation think it never ever changes." / Twitter

kornelski/http-cache-semantics: RFC 7234 in JavaScript. Parses HTTP headers to correctly compute cacheability of responses, even in complex cases

Content language

lang HTML attribute indicates the language of the text, whereas Content-Language provides metadata about the intended audience of the document.

Types of language declaration

Request content encoding

See Content encoding

Send compressed request body

If the server doesn't support the given encoding, should response with the status 415 (Unsupported Media Type)

Transfert encoding

Note that chunked is always acceptable for HTTP/1.1 recipients

About ranges and chunked transfert encoding:

Send binary data in JSON

See Request content encoding

base64 or multipart/form-data + use a "pointer" that data field name

Prevent form replay

Always provide a way for the user to keep provided values, in case the submit form data fail. Show an error message and fill fields with provided values: <input value="Value provided by the user that is not used until the error is fixed">

WebRTC

Janus WebRTC Gateway - WebRTC protocol used in client-server architecture
serverless-webrtc - A demo of using WebRTC with no signaling server
https://github.com/js-platform/node-webrtc
https://github.com/mappum/electron-webrtc

Measure network quality

https://docs.google.com/document/d/1eBix6HvKSXihGhSbhd3Ld7AGTMXGfqXleu1XKSFtKWQ/edit

Get client IP

Or proxy's IP

Note: X-Forwarded-For header can be spoofed, see The perils of the “real” client IP | adam-p

<?php
// Note: X-Forwarded-For contains a list of IP address, where the first one is the client's IP
if (array_key_exists('HTTP_CLIENT_IP', $_SERVER)) {
   $ipaddress = $_SERVER['HTTP_CLIENT_IP'];
} else if (array_key_exists('HTTP_X_FORWARDED_FOR', $_SERVER)) {
   $ipaddress = $_SERVER['HTTP_X_FORWARDED_FOR'];
} else if (array_key_exists('HTTP_X_FORWARDED', $_SERVER)) {
   $ipaddress = $_SERVER['HTTP_X_FORWARDED'];
} else if (array_key_exists('HTTP_FORWARDED_FOR', $_SERVER)) {
   $ipaddress = $_SERVER['HTTP_FORWARDED_FOR'];
} else if (array_key_exists('HTTP_FORWARDED', $_SERVER)) {
   $ipaddress = $_SERVER['HTTP_FORWARDED'];
} else if (array_key_exists('REMOTE_ADDR', $_SERVER)) {
   $ipaddress = $_SERVER['REMOTE_ADDR'];
} else {
   $ipaddress = null;// not known
}

$_SERVER['HTTP_X_REAL_IP'];

$iptype = 'unknown';
if(!empty($iptype)){
   if(preg_match("/^[0-9a-f]{1,4}:([0-9a-f]{0,4}:){1,6}[0-9a-f]{1,4}$/", $ip) != 0){
   	$iptype = 'IPv6';
   }else if(preg_match("/^([0-9]{1,3}\.){3}[0-9]{1,3}$/", $ip) != 0){
   	$iptype = 'IPv4';
   }
}
?>

apache - How to set REMOTE_ADDR to HTTP_X_REAL_IP? - Stack Overflow

Geo IP

apilayer/geolocationapi: IP Geolocation API is a free service for locating your visitors in real-time with detailed country information.
IP Geolocation API [ip-api]
The Google Maps Geolocation API | Google Maps Geolocation API | Google Developers
freegeoip.net
http://dev.maxmind.com/geoip/legacy/downloadable/
https://countryblocklist.com/
http://www.ipdeny.com/ipblocks/data/countries/

Proxy

Proxy can transform content

Can transform content: remove comments (HTML, JS, CSS), recompress (reduce JPEG quality), block files, change formats (APNG → PNG, remove fdAT and fcTL chunks), rewrite links (src or href attributes), etc.

(SFR) if file header looklike an image header (JPEG or PNG), the following content could be broken

Mobile network providers/ISP like SFR (FR), Bouygues Telecom (FR), O2 (UK), Centertel, China Mobile, Cingular, Connex, Nextel, NTT DoCoMo, Orange, Proximus, Sprint Nextel, T-Mobile, Telefónica Móviles, O2, Vodafone (IE, DE), Willcom... often use Citrix ByteMobile Proxy
Cache proxies
Proxy browsers like Opera Mini (use Citrix ByteMobile Proxy too), UCBrowser or Puffin. "reduce data usage" mode in mobile Chrome.
Other Proxies/VPN like , , , Google

To prevent proxys from modifying the website's content, serve the content with this HTTP header:

Cache-Control: no-transform

Apache config (ex.: in .htaccess):

<IfModule headers_module>
	Header set Cache-Control no-transform
</IfModule>

Serve HTTPS can't always resolve issues.

See also Ads / Content blockers like Focus or Adblock Plus or Anti Ad blocker https://github.com/reek/anti-adblock-killer/ See Validity of CSS class names and ids

Proxy Server

Use Apache as proxy server

Serve http://X.X.X.X:8080/pac.php

(Listen 8080) Some browsers map internaly localhost without proxy. To skip that, add the FQDN trailing dot to it (IE7+): localhost.

<proxy *> or <proxy http://www.google.com>

proxy.conf:

# Forward proxy server
<VirtualHost *:8080>
	ProxyRequests On
	ProxyVia On

	<Proxy *>
		Order deny,allow
		Deny from all
		Allow from 127.0.0.1
		Allow from 192.168

		RewriteEngine on
		RewriteCond %{REQUEST_URI} !/pac.php
		RewriteRule ^ /endpoint.php [L]
	</Proxy>
</VirtualHost>

endpoint.php:

<?php
// Don't handle domain existance
$url = $_SERVER['REQUEST_URI'];
$url_parts = parse_url($url);
// Some security checks (no local file...)
if(false === $url_parts || empty($url_parts['scheme']) || !in_array($url_parts['scheme'], array('http', 'https'))){
	die();
}

$headers_raw = '';
foreach ($_SERVER as $name => $value)
{
	if (substr($name, 0, 5) == 'HTTP_')
	{
		$name = str_replace(' ', '-', ucwords(strtolower(str_replace('_', ' ', substr($name, 5)))));
		$headers_raw .= $name . ': ' . $value . "\r\n";
	} else if ($name == "CONTENT_TYPE") {
		$headers_raw .= 'Content-Type: ' . $value . "\r\n";
	} else if ($name == "CONTENT_LENGTH") {
		$headers_raw .= 'Content-Length: ' . $value . "\r\n";
	}
}

// http://php.net/manual/en/context.http.php
$context = stream_context_create(array(
	'http' => array(
		'method' => $_SERVER['REQUEST_METHOD'],
		'header' => $headers_raw,
		'ignore_errors' => true,
		'content' => file_get_contents('php://input')
	)
));
$content = file_get_contents($url, false, $context);

$content_type = 'application/octet-stream';// "text/html"
$content_type_raw = $content_type;// "text/html; charset=UTF-8"
foreach($http_response_header as $response_header){
	if('Content-Type:' == substr($response_header, 0, 13)){
		$content_type_raw = substr($response_header, 14);
		$content_type = strstr($content_type_raw, ';', true);
		header($response_header);
	}
	elseif('Content-Encoding:' == substr($response_header, 0, 17) && 'gzip' == substr($response_header, 18, 4))
	{
		//Now lets uncompress the compressed data
		$content = gzinflate(substr($content, 10, -8));
	}
	elseif('Content-Length:' == substr($response_header, 0, 15))
	{
		//Skip it
	}
	else{
		header($response_header);
	}
}

// Content transforms
//var_dump($url_parts);exit();
if('text/html' == $content_type){
	echo str_replace('cat', 'dog', $content);
}else{
	echo $content;
}
?>

pac.php:

<?php header('Content-Type: application/x-javascript-config') ?>
function FindProxyForURL(url, host)
{
	 return "PROXY <?php echo $_SERVER['SERVER_ADDR'] ?>:<?php echo $_SERVER['SERVER_PORT'] ?>; DIRECT";
}

Internet Content Adaptation Protocol

Aka ICAP

Internet Content Adaptation Protocol - Wikipedia, the free encyclopedia
http://wiki.squid-cache.org/Features/ICAP
php-icap-server - Internet Content Adaptation Protocol Server implemented in PHP - Google Project Hosting
icapjs/icap_server.js at master · LordEidi/icapjs

HTTP/2.0

Performance testing HTTP/1.1 vs HTTP/2 vs HTTP/2 + Server Push for REST APIs

Header compression

Aka HPACK

Anonymous requests require clean connection

Requests without credentials use a separate connection — HTTP/2 push is tougher than I thought - JakeArchibald.com

Fonts require to be loaded as CORS anonymous or require CORS authorization: CSS Fonts Module Level 3

CORS Anonymous requests complicates preconnect · Issue #32 · w3c/resource-hints
Chrome prefix group_name for SOCKET->TRANSPORT_CONNECT_JOB group_name as pm/ (example of pm/ssl/example.com:443 vs ssl/example.com:443, pm/example.com:80 vs example.com:80) for anonymous connections chrome://net-internals/#events&q=type:SOCKET,TRANSPORT_CONNECT_JOB

Shared connection with multiple hosts

Aka connection coalescing

Multiple hosts can use the same connection, but need the same IP (?) and (when HTTPS) require to use the same certificate (that use Subject Alternative Names) But may not support this.

HTTP/2 supports multi-host multiplexing which further reduces RTTs. SPDY only supports single-host multiplexing. — HTTP/2 - What Does it Mean for a CDN?

HTTP/3

URI/URL

Cool URIs don't change
What makes a cool URI? A cool URI is one which does not change. What sorts of URI change? URIs don't change: people change them.
— Hypertext Style: Cool URIs don't change.

URI Template format definition: RFC 6570 - URI Template

absolute URL: http://domain/path/to/some/resource https://url.spec.whatwg.org/#absolute-url-string
scheme-relative (or Network-path reference or protocol-relative) URL : //domain/path/to/some/resource https://url.spec.whatwg.org/#scheme-relative-special-url-string It usage is discouraged: The protocol-relative URL - Paul Irish
absolute-path-relative URL: /path/to/some/resource https://url.spec.whatwg.org/#path-absolute-url-string
path-relative URL: path/to/some/resource, ../path/to/some/resource https://url.spec.whatwg.org/#path-relative-scheme-less-url-string
origin-form: path and query string of the URI. The query string may or may not be present.
absolute-form: an absolute URI.
authority-form: the authority part of a URI, made up of a maximum of 3 parts – user-info (optional), host and port (optional). The user-info may require a password too – user:password. We end up with a pattern of user:password@host:port. The user-info may also have require an
asterisk-form: just the string, *
same origin: same scheme, host, and port. and
same origin-domain:
same site: same registrable domain.

scheme:[//[user:password@]host[:port]][/]path[?query][#fragment]

How many ways can you slice a URL and name the pieces? - Tantek
http://[::1]:80/, http://[::]:80/, http://0.0.0.0:80/, http://127.0.0.1/, http://127.0.0.1/, http://127.1/, http://127.40.89.34/ (loopback too), http://localhost/, http://3232235778/ (see )
mysql://user:pass@host/mydatabase
smtps://username:password@smtp.example.com/?pool=true
memcached://localhost:11211/meta
redis://user:secret@localhost:6379/0?foo=bar&qux=baz https://www.iana.org/assignments/uri-schemes/prov/redis
https://github.com/sidorares/node-mysql2/blob/9218f055ceeb95ae7205348e06c07b89b799d031/lib/connection_config.js#L88-L118

Note: https://-inevitablelament.tumblr.com/ is a valid URL (subdomain start with minus char), see Bug #668926 “can't resolve domain names starting with a dash (mi...” : Bugs : resolvconf package : Ubuntu

IP address representation

aka IP address as number, aka IP decimal notation

1.1.1 will interpret to 1.1.0.1 and not 1.1.1.0 or 0.1.1.1

http://3232235778/ and http://192.168.1.2/ are equivalent http://2130706433/ and http://127.0.0.1/ are equivalent

<?php
function ipv4ToNumberumber($ip) {
	list($a, $b, $c, $d) = explode('.', $ip);
	return (double) ($a*16777216)+($b*65536)+($c*256)+($d);
}
function numberToIPv4($number) {
	$a = ($number/16777216)%256;
	$b = ($number/65536)%256;
	$c = ($number/256)%256;
	$d = ($number)%256;
	return $a.'.'.$b.'.'.$c.'.'.$d;
}
?>

Scheme-relative URLs

Aka protocol-relative URLs.

It's recommended to always use https:// URLs.

This not work if the protocol is not the same. Example: in email this URLs are not supported

Stop Using the Protocol-relative URL - Jeremy Wagner
The protocol-relative URL - Paul Irish
Protocol Relative URLs, Why to Use Them, and Common Pitfalls - Bill Patrianakos
Wikipedia:Protocol-relative URL — Wikipedia
http://tools.ietf.org/html/rfc3986#section-4.2
https://url.spec.whatwg.org/#concept-scheme-relative-url

Don't use it in email (could be resolve as file protocol: file:///...)

Domain

Naked domain = domain without subdomain (eg. www.). Aka apex domain, base, bare, naked, root apex, or zone apex

|--------------------|-----------------------|-------------------|

Domain

Registrable domain

Public suffix

edition.cnn.com

cnn.com

com

money.cnn.com

cnn.com

com

w3c.github.io

github.io

Note: public suffix aka effective top-level domain (eTDL).

www. is not deprecated – really and Why use www? | Hacker News
Chrome hide "trivial subdomains" (www., m.)
- Chrome 69 Removing WWW and M subdomains From the Browser's Address Bar
browser autocomplete/autofill user entering URLs
- browser.fixup.alternate.*, browser.fixup.*
Suffixes, like .com, .co.uk, pvt.k12.ma.us, etc.

Second-level domain

Second-level domain — Wikipedia
https://gist.github.com/rubengersons/0ff87fd457c958a503ab
https://gist.github.com/danallison/905746161b23832b9cd8029ca6919b51
https://github.com/projectbtle/BLE-GUUIDE/blob/10fdd3f3fa7ca077e0ba5a3efa2c2ac83455cb1a/resources/common/slds.json
https://github.com/gavingmiller/second-level-domains
https://github.com/medialize/URI.js/blob/master/src/SecondLevelDomains.js
https://github.com/adblockplus/adblockpluscore/blob/next/lib/url.js + https://github.com/adblockplus/adblockpluscore/blob/next/build/updatepsl.js / https://github.com/adblockplus/adblockpluscore/blob/next/data/publicSuffixList.json

Password

HTTP Authentification

http://user:password@example.com http://user:p%40ssword@example.com (user, p@ssword)

Encode special chars: mypass#gh647 to mypass%23gh647

Write URI well

See RESTful and API

Good Practice for Capability URLs
Use Canonical URLs, Please
Benefits of Plain English URLs
The Peril of Self-Replicating Hyperlinks
Slug
https://restpatterns.mindtouch.us/Articles/Designing_URIs

URI template

https://example.com/{file}
https://example.com/search{?q*,lang}

Subordinate resource

Aka subresource, hash, fragment identifier

Selectors:

RFC 3236 - The 'application/xhtml+xml' Media Type - HTML: page.html#namedSection
RFC 3778 - The application/pdf Media Type - PDF: slides.pdf#page=10&viewrect=50,50,640,480
RFC 5147 - URI Fragment Identifiers for the text/plain Media Type - Plain Text: text.txt#char=0,10, text.txt#line=10,20
- XML: page.xml#xpointer(/a/b/c), page.html#xpointer(string-range(//,"target text"))
- RDF/XML: data.rdf#namedResource
- CSV: data.csv#row=5-7
- Media: image.jpg#xywh=50,50,640,480, video.ogv#t=60,100
- SVG: image.svg#svgView(viewBox(50,50,640,480))
- EPUB3: book.epub#epubcfi(/6/4[chap01ref]!/4[body01]/10[para05]/3:10)
- Media, Text, RDF/HTML/XML: page.html#selector(type=CssSelector,value=%23elemid%20>%20.elemclass%20+%20p)
- page.html#css(.myclass)
- Github repo github.com/npm/cli#semver:^5.0
- page.html##text
- HTML/XML page.html#h5s1,2
- page.html#targetText=percent%20encoded
- page.html##some+text+reference
- page.html#:~:text=[prefix-,]textStart[,textEnd][,-suffix]
- JSON Pointer: data.json#/foo/0
jar:https://domain.com/path/to/jar.jar!/Pictures/a.jpg (jar:<url>!/{entry})
zip:password@http://domain.com/path/to/file#/Pictures/a.jpg
swf:http://domain.com/path/to/file.swf#com.domain.ClassName and http://domain.com/path/to/file.swc#com.domain.ClassName
- HTML/XML: page.html#xpath:/html/body/p[3].

More infos:

Data URI

On some browser, data URI are considered from different domain. See javascript - Programmatically accessing an iframe that uses a data URI as a source - Stack Overflow.

application/font-woff

data:application/pdf;name=document.pdf;base64,BASE64_DATA_ENCODED

Firefox require hash to be encoded/escaped
IE require more chars to be encoded/escaped
CSS # Sprites vs Data URI
Formats and protocols/URI

Data URI base 64 encoding

See Base64

Works with IE 9+ and others browsers

background: red url("data:image/svg+xml;base64,PHN2ZyB4bWxucz0naHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmcnIHZpZXdCb3g9JzAgMCAxMDAgMTAwJz48cmVjdCBmaWxsPSdncmVlbicgd2lkdGg9JzEwMCcgaGVpZ2h0PScxMDAnLz48L3N2Zz4=");

Data URI percentage encoding

Using encodeURIComponent()

Escapes all characters (including UTF-8 sequences) except the following: alphabetic, decimal digits, -, _, ., !, ~, *, ', (, ).

Note:

For data URI contains SGML like documents, (if it's allowed) you can use simple quote instead of double quotes for attributes values or mix both. This reduce the number of chars used by encoding 1 (') vs. 3 (%22). If the data URI must be quoted, double quote can be used with additional encoding: Example with CSS: use url("data:...value%3D'data'") with double quotes, simple quotes are not escaped. See xhtml - How to properly escape quotes inside html attributes? - Stack Overflow

It's possible to not encode all chars.

Some browsers require some encoded chars:

Firefox require # to be encoded
IE 11 & Edge require % to be encoded (no invalid escape sequences, eg. width='100%' , must be height='100%25' instead)

It's safe to keep unencoded:

(space, %20)
= (%3D)
: (%3A)
/ (%2F)

For security reasons, data URIs are restricted to downloaded resources. Data URIs cannot be used for navigation, for scripting, or to populate frame or iframe elements.
Data URIs cannot be larger than 32,768 characters.
The resource data must be properly encoded; otherwise, an error occurs and the resource is not loaded. The "#" and "%" characters must be encoded, as well as control characters, non-US ASCII characters, and multibyte characters. — data Protocol

copy("data:image/svg+xml," + encodeURI(`<svg xmlns="...</svg>`.replace(/"/g, "'").replace(/(^|>)\s+(<|$)/g, "$1$2")).replace(/%20/g, " ").replace(/#/g, "%23"));

MHTML URI

Multipart document

IE?-8 only

/*
Content-Type: multipart/related; boundary="_ANY_STRING_WILL_DO_AS_A_SEPARATOR"

--_ANY_STRING_WILL_DO_AS_A_SEPARATOR
Content-Location:locoloco
Content-Transfer-Encoding:base64

iVBORw0KGgoAAAANSUhEUgAAAAQAAAADCAIAAAA7ljmRAAAAGElEQVQIW2P4DwcMDAxAfBvMAhEQMYgcACEHG8ELxtbPAAAAAElFTkSuQmCC
--_ANY_STRING_WILL_DO_AS_A_SEPARATOR
Content-Location:polloloco
Content-Transfer-Encoding:base64

iVBORw0KGgoAAAANSUhEUgAAABkAAAAUBAMAAACKWYuOAAAAMFBMVEX///92dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnYvD4PNAAAAD3RSTlMAACTkfhvbh3iEewTtxBIFliR3AAAAUklEQVQY02NgIBMwijgKCgrAef5fkHnz/y9E4kn+/4XEE6z/34jEE///A4knev7zAwQv7L8RQk40/7MiggeUQpjJff+zIpINykbIbhFSROIRDQAWUhW2oXLWAQAAAABJRU5ErkJggg==
*/

#test1 {
	background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAQAAAADCAIAAAA7ljmRAAAAGElEQVQIW2P4DwcMDAxAfBvMAhEQMYgcACEHG8ELxtbPAAAAAElFTkSuQmCC"); /* normal */
	*background-image: url(mhtml:http://phpied.com/files/mhtml/mhtml.css!locoloco); /* IE < 8 */
}

#test2 {
	background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABkAAAAUBAMAAACKWYuOAAAAMFBMVEX///92dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnYvD4PNAAAAD3RSTlMAACTkfhvbh3iEewTtxBIFliR3AAAAUklEQVQY02NgIBMwijgKCgrAef5fkHnz/y9E4kn+/4XEE6z/34jEE///A4knev7zAwQv7L8RQk40/7MiggeUQpjJff+zIpINykbIbhFSROIRDQAWUhW2oXLWAQAAAABJRU5ErkJggg=="); /* normal */
	*background-image: url(mhtml:http://phpied.com/files/mhtml/mhtml.css!polloloco); /* IE < 8 */
}

`mailto:` URI

See mailto

Note: Be carefull with non encoded chars. Chrome don't open mailto url if not all required chars that require encoding are not.

You can open mailto in _self or _blank window. If an protocol handler is enable to mailto: this should open the related page (in self or blank, based on the target window).

as link <a href="mailto:?subject=Hello&body=Hello%20world!" target="_blank">maitlo</a>
as script window.open("mailto:?subject=Hello&body=Hello%20World!", "_blank")
as form <form action="mailto:" target="_blank"><input type="hidden" name="subject" value="hello"><input type="hidden" name="body" value="Hello world!"><button>mailto</button></form>. It's not a usefull method for the following reasons:
- with GET, spaces fields name and values will be replace with +.
- for file fields (<input type="file">), the value will be replace with the name of the selected file
- with POST and with encoding type text/plain, the body will contains all fields: one per line with

Target can be omited (default to _self).

`tel:` URI

See tel

Well known

Aka /.well-known/

Site-wide metadata files

Well-Known URIs
www.mysite.com/robots.txt symlink to www.mysite.com/.well-known/robots.txt (only 301 redirection are recommended, see robots.txt)
App Search Programming Guide: Support Universal Links - /.well-known/apple-app-site-association
- /.well-known/assetlinks.json
- https://<domain>/.well-known/webfinger?resource=<resource> (where resource value - URI encoded - is acct:<email-address>)

About URI

about:blank
about:invalid
about URI scheme - Wikipedia

HTTP API

RESTful

Aka REST, Hypermedia API

CRUD:

create: POST request, response with 201 Created / 202 Accepted + Location: <URI to created resource> or 303 See Other + Location: <URI to related resource>
read (retrieve): GET request, response with 200 OK
update (modify, replace): PUT request, response with 200 OK
update (modify, partial): PATCH request, response with 200 OK
delete (destroy): DELETE request, response with 204 No Content

Could use 422 Unprocessable Entity instead of 400 Bad Request when the request body is valid (syntax is valid), but the server is unable to process the instruction. See rest - 400 vs 422 response to POST of data - Stack Overflow

Patch

PATCH /my/data HTTP/1.1
Host: example.org
Content-Length: 326
Content-Type: application/json-patch+json
If-Match: "abc123"

[
 { "op": "test", "path": "/a/b/c", "value": "foo" },
 { "op": "remove", "path": "/a/b/c" },
 { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
 { "op": "replace", "path": "/a/b/c", "value": 42 },
 { "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
 { "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]

Documentation formats

Aka specs, docs format

And generators

OpenAPI
- Home - OpenAPI Initiative
- Swagger 2.0
- - "OpenAPI = Specification, Swagger = Tools for implementing the specification"
- some implementations:
Postman collection and environnements files (JSON)
https://github.com/nelmio/NelmioApiDocBundle

Hypermedia

JSON API

JSON:API — A specification for building APIs in JSON

API examples

http://lifeforms.org/<alias> redirect to (305) http://lifeforms.org/<Kingdom>/<Phylum>/<Class>/<Order>/<Family>/<Genus>/<Species>

RESTBase docs - REST API - MediaWiki
API References and Guides - PayPal Developer
GitHub API v3 | GitHub Developer Guide and https://api.github.com/

Security

There is no session

Implementation / libs

WebHook

Aka postback, pixel

Webhook - Wikipedia
WebSub - Open protocol for distributed pub–sub communication on the internet
WebSub Rocks!

Cookies

For document.cookies, max-age is not suppoted by IE and some Edge versions (it considered as session cookie)

max-age=0, expires=Thu, 01 Jan 1970 00:00:00 GMT"

document.cookie = 'name=;path=/;domain=.domain.example;expires=Thu, 01 Jan 1970 00:00:00 GMT';

// Clear cookie
// Note the case sensitive
document.cookie = "{cookie-name}=;domain={domain};expires=Thu, 01 Jan 1970 00:00:00 GMT;max-age=0;path=/";

But max-age is not well supported, see [Cookie max age](#Cookie max age)

RFC 6265 says that "Expires",* "Max-Age", "Domain", "Path", "Secure", "HttpOnly" should match case-insensitively:

there is no assertion there about the cookie name itself - there's a reason for that, because the cookie name should not be matched case insensitively, hence why it is not mentioned there.

In some browsers, the path is case sensitive

name=value

DNS

Encrypted DNS

DNS records

Local gTLD

HTTP

Headers

Use right status

Redirects can break POST

HTTPS (TLS)

Streaming and Push

Content type

HSTS

Server Push

Force download

Access

Referer

Method

Simple HTTP Server

Content encoding

Cache

Content language

Request content encoding

Transfert encoding

Send binary data in JSON

Prevent form replay

WebRTC

Measure network quality

Get client IP

Geo IP

Proxy

HTTP/2.0

HTTP/3

URI/URL

IP address representation

Scheme-relative URLs

Domain

Password

Write URI well

URI template

Subordinate resource

Data URI

MHTML URI

mailto: URI

tel: URI

Well known

About URI

HTTP API

RESTful

WebHook

Cookies

Cookie max age

Delete a cookie

Cookie names are case-sensitive

Redirects can break `POST`

`mailto:` URI

`tel:` URI