Web

The web is not one platform. It is a multitude of platforms, most of which you’ll never test on. — Peter-Paul Koch

See also:

News

PDF

Media type

Aka MIME type, content type

Some subtypes support optional parametersimage/webp;animation=y, image/webp;version=0.3

Charset case

Note: utf-8 or UTF-8? (Don't use UTF8 nor utf8). Prefer UTF-8. It's case insensitive, stick to one.

Multipart resource

Works only for images (used by MJPEG images):

Content-Type: multipart/x-mixed-replace;boundary=xxboundaryxx

--xxboundaryxx

Content-Type: image/jpeg
Content-Length: 1000

JIFFdata..frame1.1000

--xxboundaryxx

Content-Type: image/jpeg
Content-Length: 1000

JIFFdata..frame2.1000

--xxboundaryxx

It's used also for form upload with multipart/form-data used only by HTTP requests

Multiple Choices

HTTP/1.1 300 Multiple Choices
Link: <http://example.org/books/1234>; rel="alternate"; type="application/hal+json", <http://example.org/books/1234>; rel="alternate"; type="application/hal+xml", <http://example.org/books/1234>; rel="alternate"; type="application/atom+xml;type=entry"
HTTP/1.1 300 Multiple Choices
Content-type: text/uri-list

# metadata
URI1
# metadata
URI2

External-Body type

Note: It's used by emails

Set Content-Type to:

message/external-body; access-type=URL; URL="http://www.foo.com/file"
message/external-body; access-type=local-file; name="file:/local/path/file.html"

Server environnements

  1. local

  2. development (or testing, integration) (could have a dedicated env. for Quality Assurance and User acceptance Testing)

  3. staging, uat, preprod

  4. production

Detection

Always use feature detection (Don't sniff User Agent! Never!) instead of browser/platform detection. Easyer client side with JavaScript or CSS.

var webkit = !document.uniqueID && !window.opera && !window.globalStorage: old code that evaluates to true in IE, Opera + Firefox today. — Mike Taylor on Twitter

# From https://speakerd.s3.amazonaws.com/presentations/0c3a0cee177346fcb5acbe81a440377d/slide_21.jpg?1476842385
<IfModule mod_rewrite.c>
	RewriteEngine on

	# See https://github.com/serbanghita/Mobile-Detect

	# Check for mobile
	# Android is only being used below 2.3 and "Mobile" above to ignore android tablets
	RewriteCond %{HTTP_USER_AGENT} "iPhone|Android|BlackBerry|Windows CE|IEMobile|Alcatel-OT|Opera Mobi|Opera Mini|DoCoMo.*N905i|DoCoMo.*P900i|Obigo|Wapsilon|WinCE|YourWap|Motorola|Samsung-|SAMSUNG-|portalmm|Telit_Mobile_Terminals|MobilePhone|SIE-|SCH-|GoodAccess|NEC-|Mitsu|Ericsson|AUDIOVOX-|LGE-|Sanyo-|Nokia|Alcatel-B|KDDI-|WAPman|WapTunner|WinWAP|ACS-NF\/3.0 NEC-|Vodafone|ZTE-|LG-|Sendo|SHARP-|TSM-|SAGEM-|MOT-|WIG Browser|Amoi|Symbian|J2ME|HTC|SPV|MOTO|Moto|SEC-SGH|Xda|XDA|iPAQ|MDA-vario|MDA Vario|MDA Touch|MDA_Vario|MDA_compact|MDA compact|MDA_Touch|MDA_vario|Novarra|webOS"
	RewriteRule .* - [E=device:simplephone]

	# Check for smartphones
	# WebKit: iPhone 4+ (& iPod), Android 2.3+, Blackberry 6+
	RewriteCond %{HTTP_USER_AGENT} "((iPhone(.* )OS( +)(([4-9])|([0-9]+[0-9]))[^0-9])|(Android( *)((2[^0-9][1-9])|([4-9][^0-9])|([0-9]+[0-9][^0-9])))|(BlackBerry( *)(([6-9])|([0-9]+[0-9]))[^8-9]))(.*)AppleWebKit"
	RewriteRule .* - [E=device:smartphone]

	# Check for devices that look like phones but are really tablets
	# That's Android 3.x - all of which are tablets
	RewriteCond %{HTTP_USER_AGENT} "Android( ?)3\."
	RewriteRule .* - [E=device:default]

	# Override settings by cookie for device switchers
	RewriteCond %{HTTP_COOKIE} device=([^;]+) [NC]
	RewriteRule .* - [E=device:%1]

	# Override setting with query string for testing
	RewriteCond %{QUERY_STRING} "^(.+&)?device=([^&]+)(&.+)?"
	RewriteRule .* - [E=device:%2]

	# Set environment variable mobile=1 in case smart of simple phone
	# Se query string fragment: qs4device in case smart or simple phone
	RewriteCond %{ENV:device} ^(.*phone)$
	RewriteRule .* - [E=mobile:1,E=qs4device:device\=%1]

	# Otherwise it's desktop
	RewriteCond %{ENV:device} "^(default)?$"
	RewriteRule .* - [E=device:default,E=qs4device:layout=desktop]
	</ifModule>

	# Setting the Vary: User-Agent
	# Make sure proxies and google crawl the pages with different UserAgents
	# See also: https://developers.google.com/webmasters/smartphone-sites/details
<ifModule mod_headers.c>
	Header set Vary "User-Agent"
</ifModule>

Maps

Analytics

Tracking and analytics

Use Navigator.sendBeacon() - Web APIs | MDN

Tracking parameters

Aka query stripping

  • mc_eid

  • oly_anon_id

  • oly_enc_id

  • __s

  • vero_id

  • _hsenc

  • mkt_tok

  • fbclid

http {
  # redirect map in http block - remove fbclid argument from the end
  map $request_uri $redirect_fbclid {
    "~^(.*?)([?&]fbclid=[a-zA-Z0-9_-]+)$"  $1;
  }

  server {
    # if redirect map is active, do 301 to the new url
    if ( $redirect_fbclid ) {
      return 301 $redirect_fbclid;
    }
  }
}

Data layer

Customer Experience Digital Data Layer (CEDDL): This is a data layer specification developed by the World Wide Web Consortium (W3C). It is declared as a key-value JavaScript object. However, it has pretty much been discarded/deprecated/ignored because of its static interface that does not work well with modern web frameworks.

Event-Driven Data Layer (EDDL): A general term referring to any data layer that is designed to respond to data that is pushed into it, where each push is termed an "event". There's a good write-up about this at https://jimalytics.com/tag-management/the-event-driven-data-layer/. Google Tag Manager's dataLayer is a common implementation of a EDDL.

Adobe Client Data Layer (ACDL): Adobe's implementation of a EDDL. It has been integrated into AEM and some Adobe products, but can be used as a standalone data layer in any website by including its JavaScript library. See https://github.com/adobe/adobe-client-data-layer for more info.

Direct Call Rule (DCR): This is a feature that is specific to Adobe Launch (or what is now called Adobe Experience Platform Data Collection Tags). By running _satellite.track("identifier") anywhere in a web page, you can trigger a Rule in Adobe Launch that listens for that Direct Call event with the "identifier" value. You can read more about how to use it at https://experienceleague.adobe.com/docs/experience-platform/tags/extensions/adobe/core/overview.html?lang=en#other-events

— Hi, can any one help me to understand the differen... - Adobe Experience League Community - 438861

Adobe Launch

Tracking data

  • page view

  • printing

    <link rel="stylesheet" href="track_printing.css" media="print" type="text/css" />

    track_printing.css (should be served with header that disable cache):

    body { background: url("http://path.to/tracker_script"); }

  • default font size (use blank iframe)

  • default zoom level

  • screen size

  • language

  • user timing

Tracking events

  • Google Analytics Classic is repreciated

  • https://developers.google.com/analytics/devguides/collection/analyticsjs/

  • GA Event Tracking Code Generator

  • https://support.google.com/analytics/answer/1033068?hl=en

  • https://developers.google.com/analytics/devguides/collection/analyticsjs/events

  • https://developers.google.com/analytics/devguides/collection/analyticsjs/field-reference

  • https://developers.google.com/analytics/devguides/collection/analyticsjs/command-queue-reference

Be carefull of types (integer for eventValue)

Google Analytics Legacy

Aka ga.js

Use analytics.js instead

Use async, via _gaq.push()

<script type="text/javascript">
	(window._gaq = _gaq || []).push(
		["_setAccount", "UA-XXXXX-X"],
		["_trackPageview"]
	);
	(function(d,t,s,l) {s=d.createElement(t),l=d.getElementsByTagName(t)[0];s.async=true;s.src="http"+("https:"==document.location.protocol?"s://ssl":"://www")+".google-analytics.com/ga.js";l.parentNode.insertBefore(s,l);})(document, "script");
</script>
_gaq.push(["_trackEvent", category, action, label/*optional*/, value/*integer, optional*/, opt_noninteraction/*boolean, optional*/]);

Universal Analytics tracking library

Aka analytics.js

Universal Analytics tracking library (analytics.js):

ga("send", "event", category, action, label/*optional*/, value/*integer, optional*/, noninteraction/*boolean, optional*/);
ga("send", "event", category, action, label/*optional*/, value/*integer, optional*/, {nonInteraction: noninteraction}/*boolean, optional*/);
<!-- Google Analytics -->
<script>
window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date;
ga("create", "UA-XXXXX-Y", "auto");
ga("send", "pageview");
</script>
<script async src="https://www.google-analytics.com/analytics.js"></script>
<!-- End Google Analytics -->

For a custom function name: (ex: __gaTracker)

<!-- Google Analytics -->
<script>
(function(w,n){w.GoogleAnalyticsObject=n;w[n]=w[n]||function(){(w[n].q=w[n].q||[]).push(arguments)};w[n].l=+new Date;})(window,"__gaTracker");
__gaTracker("create", "UA-XXXXX-Y", "auto");
__gaTracker("send", "pageview");
</script>
<script async src="https://www.google-analytics.com/analytics.js"></script>
<!-- End Google Analytics -->

GoogleAnalyticsObject

ga("send", "event", {
	eventCategory: "Category",
	eventAction: "Action",
	eventLabel: "Label"/*optional*/,
	eventValue: 55/*integer, optional*/,
	eventNonInteraction: 55/*integer, optional*/
});

ga("send", {
	hitType: "event",
	eventCategory: "Category",//ex: Videos
	eventAction: "Action",//ex: play
	eventLabel: ""//ex: video title
});
// Use beacon or fallback to xhr (instead of image)
ga("set", "transport", navigator.sendBeacon ? "beacon" : "xhr");

Social interactions:

Google Tag Manager

Aka GTM

Handle inclusion of libraries (third party services).

See also: Third parties

Allow predefined set of events (category + action + label + value, etc.) with macros, conditions, etc.

<script>(window.dataLayer=window.dataLayer||[]).push({'gtm.start':new Date().getTime(),event:'gtm.js'});)</script>
<script src="https://www.googletagmanager.com/gtm.js?id=GTM-XXXX" async></script>
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-XXXX" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<script>
document.addEventListener("someevent", () => {
	window.dataLayer = window.dataLayer || [];

	dataLayer.push({
		event: "some event name",
		customData: "some value"
	})
});
</script>

Privacy

Use "do not track" feature navigator.doNotTrack

function trackOutboundLink(url) {
	// check if the GA object exists and that it has initialized
	if(window.ga && ga.loaded && navigator.doNotTrack == "1") {
		// if yes, rely on GA to follow link
		ga('send', 'event', 'outbound', 'click', url, {
			'transport': 'beacon',
			'hitCallback': function(){document.location = url;}
		});
	} else {
		// if not, follow link ourselves
		document.location = url;
	}
}

CNAME cloaking

Tracking virtual pageviews

Ex: Ajax pages of Single Page Application (SPA)

_gaq.push(['_trackPageview', path/*optional*/]);

ga("set", {page: path, title: title});/*optional, if need to defined a different location and title than the current document location
ga("send", "pageview");

Tracking scripts errors

Track errors, like front end (window.onerror) or backend errors with https://developers.google.com/analytics/devguides/collection/analyticsjs/exceptions

function trackJavaScriptError(event) {
	//ga("send", "event", "JavaScript Error", event.message, `${event.filename}:${event.lineno}:${event.colno}`, {nonInteraction: 1 });

	ga("send", "exception", {
		exDescription: `${event.message} ${event.filename}:${event.lineno}:${event.colno}`,
		exFatal: true,
		appName: APP_NAME,
		appVersion: APP_VERSION
	});
	/*
	ga("send", {
		hitType: "exception",
		page: "/current-url",
		exDescription: `${event.message} ${event.filename}:${event.lineno}:${event.colno}`,
		exFatal: true,
		appName: APP_NAME,
		appVersion: APP_VERSION
	});
	*/
}

window.addEventListener("error", trackJavaScriptError, false);

  • Website Exception Monitoring Template

  • https://stackoverflow.com/questions/21718481/report-for-exceptions-from-google-analytics-analytics-js-exception-tracking

  • https://davidwalsh.name/track-errors-google-analytics

Anchor hash tag behaviour

  • scroll-margin

  • use an anchor (<span id="target"></span>) in targeted element (don't use a without href) (via position: absolute; top: -200px; etc.). Position absolute can also be used, to prevent autoscroll

  • (on click on link) event.preventDefault();, history.pushState() (window.location.hash will scroll automatically, so need to override document.body.scrollTop and document.body.scrollLeft after changing its value)

  • rename target IDs. Some browser keep in memory the position of the previous ID

  • History.scrollRestoration = "manual" https://developer.mozilla.org/en-US/docs/Web/API/History/scrollRestoration

  • http://lea.verou.me/2011/05/change-url-hash-without-page-jump/

  • http://css-tricks.com/hash-tag-links-padding/

  • Autoscroll to anchors

In frame blocking

<script>
	if (self !== top) {
		document.body.style.display = "none";
		top.location = self.location;
	}
</script>

Use header Content-Security-Policy: frame-ancestors 'self' (or the depreciated X-Frame-Options: SAMEORIGIN)

See Clickjacking

Adaptive

Aka adaptive web design, responsive, adaptative web design

Deliver the best possible experience to the widest possible audience

Device Detection: Don't Do It!

Note: Sometimes it's better to have a dedicated version for a context. Ex: mobile version

  • RWD (responsive web design) = provide an optimal viewing experience, adapts the layout to the viewing environment by using fluid, proportion-based grids, flexible images, and CSS3 media queries, an extension of the @media rule

  • Progressive enhancement = Allows everyone to access the basic content and functionality of a web page, using any browser or Internet connection, while also providing an enhanced version of the page to those with more advanced browser software or greater bandwidth

Depends on features detections and usage:

Responsive images

Aka content aware image cropping and art direction

Define an interest area or predefined crop rules based on breakpoints

See also HTML - Responsive image

art direction problem involves wanting to change the image displayed to suit different image display sizes — Responsive images - Learn web development | MDN

Service generate art directed images:

Progressive Enhancement

Aka feature detection

See offline-first approach, Progressive Enhancement (CSS)

Old browsers support

Only in the case of progressive enhancement, if it's not work and no alternative is available for a major functionality, show a message about it.

Internationalization

Aka I18n

Image sequence

Aka turn table, Object VR, 360° object, object VR

Use turntable to captures differents angles.

Similar to a video, but with the ability to play in both ways, seek, without any audio track

Methods:

  1. decompose effect and recreate it with 3D models, JS, CSS, SVG... But won't look the same

  2. use video with specific frames encoding. Use only I-Frames or B-frames (B for bidirectional frame). See Frame types

  3. use images: Can use format like APNG, MNG, JNG, Animated WebP, SVG (animation), QTVR, tar archive contains images and data files. If multiple images use the same container, content encoding could be used to compress more (GZip, Brotli) to reduce redonancy between images. For JPEG, use same DCT coefficients and omit it to reduce weight. Use codecs of compressed texture format, JPEG, PNG, etc.

    1. images diff (pixels + metadata) and draw the patched image (frames: I,P,P,P...) P images could refer to the I frame or to the previous P frame (see how APNG works) Need fine control of playback

    2. bunch of images loaded into blobs and load into the DOM (IMG or Canvas or via ImageData) and drawn on canvas (frames: I,I,I,I...)

Examples, using differents methods:

Third parties

and widgets

Don't trust your third parties

See also Third parties webperf

Comments services:

Source code embed:

Website JS plugins

To isolate a third party script (or DOM access, geolocation, modal APIs, etc.), use:

See also:

oEmbed

Web Application

Viewport size

iPhone 4/4s
iPhone 5/5c/5s/SE
iPhone 6…
iPhone 6… Plus
iPad pro 12.9
iPad pro 9.7

Screen Size[1]

320 × 480 pt

320 × 568 pt

375 × 667 pt

414 × 736 pt

1366 × 1024 pt

1024 × 768 pt

Rendered Pixels

640 × 960 (@2x)

640 × 1136 (@2x)

750 × 1334 (@2x)

1242 × 2208 (@3x)

2732 × 2048 (@2x)

2048 × 1536 (@2x)

Physical Pixels

640 × 960

640 × 1136

750 × 1334

1080 × 1920

2732 × 2048

2048 × 1536

Pixels Per Inch (PPI)

326

326

326

401

264

264

Browser viewport

-

-

-

-

-

-

- Portrait

-

-

-

-

-

-

* normal[2]

320 × 372 px

320 × 460 px

375 × 559 px

414 × 628 px

?

?

* minimal[3]

320 × 440

320 × 528

375 × 627

414 × 696

?

?

* fullscreen[4]

320 × 460

320 × 548

375 × 647

414 × 716

?

?

- Landscape

-

-

-

-

-

-

* normal[2]

480 × 212 px

568 × 212 px

667 × 267 px

736 × 306 px

?

?

* minimal[3]

480 × 280

568 × 280

667 × 335

736 × 374

?

?

* fullscreen[4]

480 × 300

568 × 300

667 × 355

736 × 394

?

?

  1. JS: screen.[height|width]; CSS: device-width, device-height

  2. normal-ui (with normal browser navigation bars) JS: window.inner[Height|Width]; CSS: width, height

  3. minimal-ui (with the small browser navigation bars) JS: window.inner[Height|Width]; CSS: width, height

  4. fullscreen (without any browser chrome for a web app, only status bar if any) JS: screen.avail[Height|Width]; CSS: width, height

Privacy

Aka GDPR or EU Cookie Law, law and privacy legislations

See also Cookies

See Do Not Track https://www.eff.org/dnt-policy navigator.doNotTrack https://en.wikipedia.org/wiki/Do_Not_Track

Cookie banner:

Cookies

See also Privacy

App Download interstitials

App Association

Open native application

App install banner

<meta name="apple-itunes-app" content="app-id=myAppStoreID, affiliate-data=myAffiliateData, app-argument=myURL">

<link rel="manifest" href="/manifest.json">
{
	"prefer_related_applications": true,
	"related_applications": [
		{
			"platform": "play",
			"id": "com.google.samples.apps.iosched"
		}
	]
}

Android app links

Android App Link: https://<domain>/.well-known/assetlinks.json

Apple app site association

Aka AASA or Universal Links

https://<domain>/apple-app-site-association or https://<domain>/.well-known/apple-app-site-association without redirect (200 OK only)

Fetch/apdated when (no expired):

  • app install

  • app update

Can't use MITM to handle/override association (certificate pinning)

Dummy and placeholder media

Aka Video test pattern, test image, sample image

See Sample files

data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='450' height='300'/%3E

16/9 image:

data:image/svg+xml,%3Csvg xmlns='w3.org/2000/svg' viewBox='0 0 16 9' width='16' height='9'%3E%3C/svg%3E
<svg version="1.1" xmlns="http://www.w3.org/2000/svg">
   <g stroke="purple" stroke-width="2">
   	<line x1="0" y1="0" x2="100%" y2="100%"/>
   	<line x1="100%" y1="0" x2="0" y2="100%"/>
   	<rect x="0" y="0" width="100%" height="100%" fill="none" stroke-width="4" />
   </g>
</svg>
<svg version="1.1" xmlns="http://www.w3.org/2000/svg">
	<g stroke="purple" stroke-width="2">
		<line x1="14.64%" y1="14.64%" x2="85.36%" y2="85.36%"/>
		<line x1="85.36%" y1="14.64%" x2="14.64%" y2="85.36%"/>
		<ellipse cx="50%" cy="50%" rx="50%" ry="50%" fill="none" stroke-width="4"/>
	</g>
</svg>

Web fonts

See CSS - Font web safe

Favicon

For icons, use a large one 180×180 icon for iOS and Android

Some browsers usally try to get the following resource: /favicon.ico

<svg xmlns="http://w3.org/2000/svg" viewBox="0 0 100 100">
    <text y=".9em" font-size="90">📦</text>
</svg>

WebIDL

.webidl, .widl, .idl

Tools:

Format:

WebIDL Sources:

See also:

Browsers

Browsers version fragmentation

Version penetration

Browser support

User Agent String

Can be used server side to detect browser, bot, etc.

Apple Shortcuts App: Shortcuts/700 CFNetwork/974.2.1 Darwin/18.0.0

See Detection

Browsers extension

Aka Web Extensions

Sideloading in Firefox is no more allowed since version 73:

Sideloading in Chrome:

Safari Extensions

Safari App Extension

Safari Extension

Depreciated by Safari 12

Use xar

xar -x -f XXXXX.safariextz

Ex: https://safari-extensions.apple.com/details/?id=com.diigo.safari.awesomescreenshot-5DXNM3K2CT

(new URL(window.ex.render.install_link(window.ex.data.getItemByID((new URL(location)).searchParams.get("id"))), document.baseURI)).href

Kiosk mode

See also:

Firefox

Awesome bar:

  • Add ^ to search for matches in your browsing history.

  • Add * to search for matches in your bookmarks.

  • Add + to search for matches in pages you've tagged.

  • Add % to search for matches in your currently open tabs.

  • Add ~ to search for matches in pages you've typed.

  • Add # to search for matches in page titles.

  • Add @ to search for matches in web addresses (URLs).

  • Add $ to search for matches in suggestions. – Awesome Bar - Search your Firefox bookmarks, history and tabs from the address bar | Firefox Help

  • Open Tabs to the right of the current tab: about:config?filter=browser.tabs.insertAfterCurrent, https://addons.mozilla.org/en-US/firefox/addon/always-right/

Source code:

Dev tools:

Network debug

Session / tabs

  1. open about:support (profile directory)

  2. open the directory

  3. open the sub directory sessionstore-backups

Files in this directory:

  • previous.jsonlz4: cleanBackup: copy of sessionstore.js from previous session that was loaded successfully

  • recovery.jsonlz4: latest version of the sessionstore written during runtime

  • recovery.baklz4: previous version of the sessionstore written during runtime

  • upgrade.jsonlz4-<build_id>: backup created during an upgrade of Firefox

See also:

Integrated authentication

Aka silent authentication

authenticate the user to an Intranet server or proxy without prompting the user for a username or password

  1. go to about:config

  2. accept harmful consequences warning

  3. search network.negotiate-auth.trusted-uris and update with value .example.com to allow auto login on that domains

  4. if required, search network.negotiate-auth.allow-non-fqdn and set to true to allow non fully qualified domain names in the previous given list

For the list of trusted URIs, see:

Safari

Source code:

Dev tools:

WebKit on Windows

From playwright (a NPM library) WebKit Windows builds:

Chrome

Source code:

Dev tools:

Skia:

Enable Chrome developer tools experiments:

  1. chrome://flags/#enable-devtools-experiments

  2. webdev tools "Settings"

  3. webdev tools "Experiment" tab

  4. press "Shift" 6 times to show more experiments

Update error "An error occurred while checking for updates: Update check failed to start (error code 4: 0x80070005 – system level)" on Windows (one of):

  • run Chrome browser as administrator

  • start Google Update Service (gupdate) (services.msc) then update start up type: Automatic or Manual

  • reinstall Chrome browser (without uninstall it)

  • in C:\Program Files\Google\Chrome\Application or C:\Program Files\Google\Chrome Beta\Application, rename new_chrome.exe to chrome.exe and new_chrome_proxy.exe to chrome_proxy.exe

  • "%USERPROFILE%\Local Settings\Application Data\Google\Chrome\Application\0.4.154.25\Installer\setup.exe" --rename-chrome-exe --verbose-logging

Network debug

NetLog log:

  • queueing delay to schedule DNS resolves to threads

  • stalls due to exceeding socket pool limits

  • attempts to do a TCP connect to an IP address

  • speculative DNS resolves

  • proxy resolution

  • cache hits for DNS resolves

  • reads/writes from disk cache

  • network change events

  • proxy configuration change events

  • stalls due to chrome extensions pausing requests

  • errors

  • cookie store

Features needing reliable network information should never be built on top of NetLog

To know what extension handle requests (marked as 307 Internal Redirect, see delegate_info, CHROME_EXTENSION_REDIRECTED_REQUEST, URL_REQUEST_FAKE_RESPONSE_HEADERS_CREATED)

const readline = require("readline");
const {createReadStream} = require("fs");

async function readNetLogFile(file, eventFilter = null){
  const readlineIterable = readline.createInterface({
    input: createReadStream(file),
    crlfDelay: Infinity,
  });

  let result = null;

  const useEventFilter = typeof eventFilter === "function";

  let parent = null;
  lineLoop: for await(const line of readlineIterable){
    switch(true){
      // header
      case line.startsWith("{\"constants\":"):{
        result = {
          ...JSON.parse(line.slice(0, -1) + "}"),
          events: [],
          polledData: null,
        };// remove trailing comma and add the closing parenthese
        break;
      }
      // start events section
      case line.startsWith("\"events\":"):
        parent = "events";
        break;
      // event
      case parent === "events":{
        const eof = line.endsWith("]}");
        const isLastEvent = eof || line.endsWith("],");
        // If it's the last event, before next section
        // then skip the events array close braket and trailing comma
        // else skip the trailing comma
        const rawData = line.slice(0, isLastEvent ? -2 : -1);
        const event = JSON.parse(rawData);

        // Filter events
        if(!useEventFilter || eventFilter(event, result)){
          result.events.push(event);
        }

        if(eof){
          parent = null;
          break lineLoop;
        }

        if(isLastEvent){
          parent = null;
        }
        break;
      }
      // footer
      case line.startsWith("\"polledData\":"):
        // TODO
        break;
      // EOF
      case line === "}":
        parent = null;
        break lineLoop;
      default:
    }
  }

  return result;
}

Custom device with higher granularity

  • https://developer.chrome.com/devtools/docs/device-mode#custom-devices

  • https://github.com/ChromeDevTools/devtools-frontend/blob/9f1eee62ea4b12b10515582647260353f2f0c5e5/front_end/models/emulation/EmulatedDevices.ts#L645-L1801 default emulated devices

  • ~/Library/Application Support/Google/Chrome/Default/Preferences JSON, data.devtools.preferences.customEmulatedDeviceList

  • chrome.experimental.devtools.*

See also:

  • https://github.com/GoogleChrome/devtools-device-data/blob/release/devices.json

Update loop

Nearly up to date! Relaunch Google Chrome to finish updating.

How a web browser works

inter-frame, event loop

Caches: ( Application DNS cache) → ( libc DNS cache ) → ( gateway DNS cache ) → ( DNS caching resolver cache ) → ( DNS nameserver authoritative cache ) → Filesystem Cache → Hard disk Device Cache → ( JavaScript JIT cache ) → HTTP cache → HTTP push cache → HTTP Preload cache → CPU L1 → CPU L2 → (CPU L3) → ( HTTP proxy cache ) → (N hops invoking many of these steps N times) → ( HTTP router/relay cache ) → ( HTTP Reverse Proxy cache ) → HTTP Server request caching. (add also CSS cache and Image cache)

See also How ECMAScript engine works (event loop, etc.)

Headless browser

See Data - Web scrapping

Headless browser detection

It's a cat-and-mouse game

Reader view

Firefox Reader view, based on Readability:

Safari reader mode, based on Readability:

  1. open safari web inspector.

  2. disable (pause) Javascript in the Safari console debugger

  3. open a web page that can be view in reader mode

See also Safari Reader Source Code and anonyme.js (ReaderArticleFinder)

Other:

"Request Desktop Site"

On mobile browsers

Links and alternate

Canonical URL = the preferred URL (same document content, aka duplicated content) Alternate URL = other version of this canonicalized document (mobile or other lang)

For languages alternate, each canonicalized document must include the list of all languages available, even itself

Use alternate URL only for canonicalzed documents

On desktop:

<link rel="alternate" media="only screen and (max-width: 640px)" hreflang="en-gb" href="http://m.example.com/page">

or via http header:

Link: <http://m.example.com/page>; rel="alternate"; hreflang="en-gb"; media="only screen and (max-width: 640px)"

And on mobile:

<link rel="canonical" href="http://www.example.com/page">

For HTTPS page (on http://www.example.com/page):

<link rel="canonical" href="https://www.example.com/page">

Date

use ISO 8601 in JSON

Date#toJSON()
date('c', time());
$date->format(DateTime::ATOM/*use it instead of ISO8601*/);
$date = new DateTime("2010-07-05T06:00:00Z");
// To offset with a timezone:
$date->setTimeZone(new DateTimeZone("Europe/Amsterdam");

Troubleshooting

  1. open Chrome

  2. open a white page:

    1. open a new tag

    2. write this URL in address bar + enter : about:blank

  3. open developement tools:

    • Crtl+Maj+I

    • or Menu at the top right (< ⁝ >), then "More tools", then "Developement tools"

    • or right clic on page, then "Inspect"

  4. enable history logs perservation:

    1. open "Console" tab

    2. open the menu "Console settings" (< ⚙️ >)

    3. check "Preserve log"

  5. enable requests history perservation:

    1. open "Network" tab

    2. check "Preserve log"

  6. follow your scenario: keep developement tools opened in background

    1. open the first page (in the browser tab by write/pasting it URL in the address bar + enter)

    2. the follow the rest of the scenario

  7. once the scenario is complete, without closing the browser tab, show the developement tools window

  8. save console logs:

    1. go to "Console" tab

    2. right click on the main area (where you see yellow, red and white entries)

    3. choose "Save as..."

  9. save requests hisoty:

    1. go to "Network" tab

    2. right click on the main area (where you see all logged entries)

    3. choose "Save all as HAR with content"

  10. send us both saved files

Cross domain & integrity

  • integrity <script> and <link> attribute

  • crossorigin <script>, <img> attribute

  • noreferrer

  • noopener

  • X-Frame-Options header (depreciated)

  • Content-Security-Policy header

See XSS and injection prevention

Subresource checksum

Aka subresource integrity

See also nonce attribute.

JavaScript:

let source = 'alert("Hello, world.");';//string
let algo = "sha256";//"sha256" / "sha384" / "sha512"
const utf8Encoder = new TextEncoder();
let sourceBytes = utf8Encoder.encode(source);
const algos = {
	sha256: "SHA-256",
	sha384: "SHA-384",
	sha512: "SHA-512"
};
crypto.subtle.digest(algos[algo], sourceBytes).then(valueBuffer => {
	let hashBytes = new Uint8Array(valueBuffer);
	let valueChars = hashBytes.reduce((chars, byte) => (chars += String.fromCharCode(byte), chars), "");// to UTF-8
	let base64Value = btoa(valueChars);
	return "'" + algo + "-" + base64Value + "'";// "'" hash-algo "-" base64-value "'"
}).then(checksum => console.log(checksum))
// Inject an inlined script with integrity
const src = `alert("hello")`;
const s = document.createElement("script");
s.src = "data:text/javascript," + encodeURI(src);
s.crossOrigin = "anonymous";
s.integrity = "sha256-" + btoa(
	String.fromCharCode(...new Uint8Array(await crypto.subtle.digest("SHA-256", new TextEncoder().encode(src))))
);
document.head.append(s);
// That works only if CSP allow data: URIsn which is a bad practice: https://security.stackexchange.com/a/95011

PHP:

$script = 'alert("Hello, world.");';
$algo = 'sha256';// 'sha384' // 'sha512'
$script_hash = base64_encode(hash($algo, $script, true));
header('Content-Security-Policy: default-src \'self\'; script-src \'self\' \''.$algo.'-'.$script_hash.'\'');// sha256-qznLcsROx4GACP2dm0UCKCzCG+HiZ1guq6ZZDob/Tng= or sha512-YWIzOWNiNzJjNDRlYzc4MTgwMDhmZDlkOWI0NTAyMjgyY2MyMWJlMWUyNjc1ODJlYWJhNjU5MGU4NmZmNGU3OAo=
echo '<script>'.$script.'</script>';

Command line (sh, openSSL):

echo sha256-$(echo -n 'alert("Hello, world.");' | openssl dgst -sha256 -binary | openssl enc -base64 | tr -d '\n')

Be carefull of (significant) spaces, quotes, etc. The hash could be different.

Bypass cross domain control

If your JS need a resource but it's disallow but the third party (does not allow with Access-Control headers)

Always filter which URL/domain you allow. Else some evil people could use your proxy for illegal activity, etc.

  • reverse proxy

  • http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%27http%3A%2F%2Fbbc.co.uk%27%0A&format=json select * from html where url='http://bbc.co.uk'

  • JSONP (previous example support &callback=foo)

  • https://github.com/Athlon1600/php-proxy

Note: CSP can block it.

Server logging

Server timing

Aka Server Timing API

Server-Timing: cache;desc="Cache Read";dur=23.2,fs;dur=600;desc="File System",cpu;dur=1230;desc="Total CPU"

CMS

JSON Schema + form:

Google Docs Sheets as CMS

Need to: File > Publish to the Web

  • https://spreadsheets.google.com/feeds/list/{worksheet_id}/{sheet_id}/public/full?alt=json

  • https://spreadsheets.google.com/feeds/worksheets/{sheet_id}/public/full?alt=json

  • https://spreadsheets.google.com/feeds/worksheets/{sheet_id}/private/full

Where sheet_id is default or 1, 2, etc. match the tab order

Web IDE

See also: zvizvi/GitHub-Web-IDE: ⚡ Open GitHub repositories in online web IDE

User script

openuserjs:

Bookmarklet

See Bookmarklet

Note: Bookmarklets are blocked by CSP in Firefox, where bookmarklet = unsale-inline script

// http://www.squarefree.com/2005/05/16/bookmarklets-to-user-scripts/
let userScriptURI = "data:text/javascript;charset=utf-8,"
	   + "// ==UserScript==%0A"
	   + "// @namespace http://www.squarefree.com/bookmarklets-to-user-scripts%0A"
	   + "// @name " + bookmarkletName + "%0A"
	   + "// @description Runs the " + bookmarkletName + " bookmarklet from " + location.href + "%0A"
	   + "// ==/UserScript==%0A"
	   + "%0A"
	   + encodeURIComponent(decodeURIComponent(bookmarklet.href))
	   + "%0A//.user.js"

copy("javascript:" + prompt().replace(/[\s%]/g, match => `%${match.charCodeAt(0).toString(16).padStart(2, "0")}`))

Accessibility

A benefit of descriptive links is that screenreader users can pull all the links on a page together into a list and find what they need. ‘Click here’ doesn’t describe where you are linking to and means nothing without textual context! Eg. Instead of “Our school has X many students and does Y and Z. For more information, click here” to link to your About Us page, use: “Our school has X many students and does Y and Z. Read more About Us” Ps. If you do this you will not only make life easier for people who use screen readers but you also have a really simple way to improve your Search Engine Optimisation. Nobody Googles “Click here.” (the WCAG 2.1: 2.4 rule - Ed.)

— Emily Smith on Twitter: "A benefit of descriptive links is that screenreader users can pull all the links on a page together into a list and find what they need. ‘Click here’ doesn’t describe where you are linking to and means nothing without textual context! (2)" / Twitter

See Accessibility

  • EU: ETSI EN 301 549, require WCAG 2.1 Level AA

  • FR: ETSI EN 301 549 and fallback to internations standards + use RGAA (based on WCAG 2.1) for technical requirements - public online communication services, public institutions, and the state

  • DE: BITV 2.0 (based on WCAG 2.1 Level AA) - government websites

  • IT: Stanca Act (based on WCAG 2.0 Level A)

  • JP: JIS X 83141 (based on WCAG 2.1)

  • ES: UNE 139803:2012 (based on WCAG 2.1 Level AA): government and government-funded organizations; or organizations larger than 100 employees; or with trading column greater than six million euros; or those providing financial, utility, travel/passenger, or retail services online

  • US: Section 508 require WCAG 2.0 Level AA

  • AU: Disability Discrimination Act (Advisory Notes, based on WCAG 2.0)

See also:

About support of old browser vs accessibiliy:

Stats:

Tools:

Other:

Screensaver

Create a screensaver from a web page:

  1. télécharger l’application « HTML Screen Saver » sur http://myweb.tiscali.co.uk/djmclean/htmlscreensaver.html

  2. l’installer l’application

  3. ouvrir le panneau de configuration (du système) (click droit sur le bureau → personaliser)

  4. ouvrir les préférences du économiseur d’écran / screen saver

  5. choisir dans la liste déroulante « HTML Screen Saver », si c’est pas déjà le cas

  6. cliquer sur options / settings

  7. parcourir et sélectionner le fichier .html précédemment reçu

  8. clicker sur Test pour prévisualiser

  9. clicker sur OK

BaaS

Aka backend as a service, Mobile backend as a service (MBaaS)

Email:

SMS and phone call (voice):

Transfert files

Digital signature

Trusted third party

Advertising

Aka Ads

See also:

A/B testing

Content blocking

Safe Browsing (UrlSubresourceFilter, BetterAds)

See also:

Safe browsing

Phishing facts:

  • phishing sites have a lifecycle of about 15 hours

  • most malicious links are hidden within benign domains

  • about 400,000 phishing sites are created each month

Code:

Datasets

Datasets:

Sourcemap

Aka source map, VLQ, variable-length quantity

Inspect:

For CSS and JavaScript

PWA

See also

Wi-Fi Captivate

Walled garden, captive portal

Tested URLs:

On iOS and OSX, when WiFi connection is detected, the OS try to open this URL: http://captive.apple.com/hotspot-detect.html (previously http://apple.com/library/test/success.html) which return 200 Ok with content <HTML><HEAD><TITLE>Success</TITLE></HEAD><BODY>Success</BODY></HTML>

  • https://stackoverflow.com/questions/18891706/ios7-and-captive-portals-changes-to-apple-request-url

  • http://blog.erratasec.com/2010/09/apples-secret-wispr-request.html

  • http://www.davidcalculli.com/blog/2012/10/apples-secret-connecting-to-a-wi-fi-network-requires-an-active-internet-connection

  • iOS Wi-Fi profiles: About Auto Join and per-connection password settings - Apple Support

  • How can I get Wifi if "http://www.apple.com/library/test/success.html" is blocked?

  • https://en.wikipedia.org/wiki/Captive_portal

  • /System/Library/CoreServices called Captive Network Assistant.app /Library/Preferences/SystemConfiguration/CaptiveNetworkSupport/Settings.plist

  • http://apple.stackexchange.com/questions/211593/how-to-disable-captive-apple-com

  • http://superuser.com/questions/937325/wifi-hotspot-why-os-does-not-detect-me-as-a-captive-portal

  • http://blog.erratasec.com/2010/09/apples-secret-wispr-request.html

  • http://www.dd-wrt.com/wiki/index.php/Chillispot

  • http://coova.github.io/CoovaChilli/

  • http://blog.trifork.com/2013/01/15/building-a-captive-portal-controlling-access-to-the-internet-from-your-network/

  • https://www.howtoforge.com/tutorial/how-to-install-a-wireless-hotspot-with-captive-page-in-linux-using-coovachilli/

  • https://learn.adafruit.com/setting-up-a-raspberry-pi-as-a-wifi-access-point/install-software

  • http://www.pihomeserver.fr/2014/05/22/raspberry-pi-home-server-creer-hot-spot-wifi-captive-portal/

  • http://www.pihomeserver.fr/2014/05/23/raspberry-pi-home-server-creer-hot-spot-wifi-portail-captif-22/

Popup

Allow multiple popup, or open popup from non user action:

  • Chrome:

    1. open website informations (from the URL bar)

    2. click website parameters (chrome://settings/content/siteDetails?site=http%3A%2F%2Fexample.com)

    3. set "Popups & redirections" to "Allowed"

  • Firefox:

    1. open page informations (Tools > Page informations)

    2. go to permissions tab

    3. uncheck "default permissions" for "Open popus" and select "Allow"

  • Edge:

    1. open Settings

    2. go to Advanced Settings

    3. switch Block Pop Ups Off

    4. after using the desired website, roll back these settings

Parties

  • first party: the host, the application

  • second party: CDN (substitution)

  • third parties

End-to-end tests

Use ID to traget elements: IDs for robots

To find an element with specific classname token foo, do not use //*[@class='foo'] nor //*[starts-with(@class,'foo')], but:

  • //*[contains-token(@class, 'foo')] (XPath 3.1)

  • //*[count(index-of(tokenize(@class), 'foo')) >= 1] (XPath 2.0)

  • //*[contains(concat(' ', normalize-space(@class), ' '), concat(' ', 'foo', ' '))] (XPath 1.0)

See also:

Website analysis

Aka technology sniffing

For:

PreviousToolsNextWebsites informations and tips

Last updated