Pre-render API

Endpoint: service.headless-render-api.com

Pre-render/server-side render Path: /$URL

Example URL: 

https://service.headless-render-api.com/https://example.com/

The target $URL is obviously part of the path, but do not re-encode or re-escape it beyond what you'd enter into a browser URL field. Both UTF-8 encoding and percent-encoding are acceptable.

Related links:

This document describes the pre-rendering functionality/endpoint, its associated HTTP header options, and possible error responses.

See the following links for related information:

Auth

Send your secret API token (you'll get it after creating an account) as a header with all your requests to avoid rate limiting.
X-Prerender-Token
  • curl --header "X-Prerender-Token: secret-token"

Pre-render a URL (i.e. pre-rendering, server-side rendering)

You can think of this endpoint almost like a proxy, and in some ways a scraper. It returns the serialized HTML of the requested URL (the "origin") - useful for converting JavaScript apps to their JavaScript + HTML representation instead of just JavaScript.

All of our integration libraries (Node, Nginx, Apache etc.) proxy your requests through this endpoint. It can also be used as a conventional API - but if used as a conventional API, be mindful that the default options are designed to favor it as a proxy endpoint (caches, doesn't follow redirects, includes some server-side rendering compatibility add-ons)

cURL

curl https://service.headless-render-api.com/https://example.com/ > out.html

Standalone Node.js pushstate server w/pre-rendering configured

// 1. build/compile your JavaScript single-page app
// 2. cd into your build/dist directory

npx prerendercloud-server

Node.js HTTP middleware

// npm install prerendercloud
const app = require('express')();
const prerendercloud = require('prerendercloud');

app.use(prerendercloud);

See more low-level examples, or see high-level pre-rendering docs.

How various content-types are handled:

  • only text/html is processed (you can't serialize a jpeg)
  • any other content-types are unprocessed and simply proxied
  • application/octet-stream fails with status code 400
  • all requests are subject to billing

Status codes

  1. 5xx from origin will return as 502 Bad Gateway (don't bother retrying)
  2. 1xx-4xx from origin are returned unmodified (including 301/302 redirects)
  3. a retryable error/timeout during headless-render-api.com's processing will return as 500 (or 503)
  4. user/request error returns as 400, rate limited as 429

Controlling status codes

  1. if your request includes the Prerender-Follow-Redirects header and the submitted URL returns 301/302, the service will follow the redirects and return final status code
  2. if the page includes a meta tag<meta name="prerender-status-code" content="xxx">, the service will return the specified code (scroll down to read more)

Optional HTTP Headers for Configure Server Cache

prerender-disable-cache
  • Disables/ignores the default 300 second (5 minutes) cache on the headless-render-api.com servers
  • curl --header "Prerender-Disable-Cache: true"
prerender-cache-duration
  • Change the default cache duration from 300s (5 minutes) to anything up to 2592000s (1 month)
    (ignored if cache is disabled)
  • curl --header "Prerender-Cache-Duration: 2592000"
prerender-recache
  • Updates/refreshes the cache even if it already exists. It will default to 300s, so use with Prerender-Cache-Duration
    (ignored if cache is disabled)
  • curl --header "Prerender-Recache: true"

Optional HTTP Headers for Disable Injected Scripts

prerender-disable-ajax-preload
  • Removes a JavaScript monkeypatch from the prerendered page that preloads your intial AJAX requests

    Read more and see  Ajax-Preload source code
  • curl --header "Prerender-Disable-Ajax-Preload: true"
prerender-disable-ajax-bypass
  • Removes a JavaScript monkeypatch from the prerendered page that adds X-Prerendered to the header of all XHR and fetch requests signaling to the middleware to go straight to origin

    Read more and see  Ajax-Bypass source code
  • curl --header "Prerender-Disable-Ajax-Bypass: true"
prerender-disable-head-dedupe
  • Removes a JavaScript monkeypatch from the prerendered page that is intended to prevent duplicate meta/title/script/style tags

    Read more and see  Head-Dedupe source code

    Some libs/frameworks detect existing meta/title/style and don't need this, but in our experience this is still a worthwhile default.
  • curl --header "Prerender-Disable-Head-Dedupe: true"

Optional HTTP Headers for Headless-Render-API Server Behavior

prerender-wait-extra-long
  • Waits longer than normal for a page to finish rendering. Similar to Puppeteer's { waitUntil: 'networkidle' }. Useful for pages that depend on AJAX/XHR that fire late or IPFS hosted pages
  • curl --header "Prerender-Wait-Extra-Long: true"
prerender-dont-wait-for-web-sockets
  • By default, headless-render-api.com will wait for all ws activity to finish, but it doesn't make sense to "wait" for them to finish if they never stop. An example: real time prices on a stock price website.
  • curl --header "prerender-dont-wait-for-web-sockets: true"
prerender-block-cookies
  • By default, headless-render-api.com sends cookies back to the server. Use this to block them.
  • curl --header "prerender-block-cookies: true"
origin-header-whitelist
  • A CSV of headers to forward. (must start with prerendercloud-) By default, the GET request headless-render-api.com makes to your origin server is clean/pure (without any remnants of the original user request). This header allows you to forward certain headers to your origin.

    Usage of this header disables the headless-browser-api server-side cache

    see more examples here
  • curl --header "origin-header-whitelist: prerendercloud-whatever,prerendercloud-b" \
        --header "prerendercloud-whatever: some-value \
        --header "prerendercloud-b: some-other-value"
prerender-meta-only
  • This option tells headless-render-api.com to only pre-render the <title> and <meta> tags in the <meta> section. The returned HTML payload will otherwise be unmodified.
  • curl --header "Prerender-Meta-Only: true"
prerender-remove-script-tags
  • Removes all script tags except for application/ld+json. If you're pre-rendering, this means no client-side JavaScript and thus your app will no longer be isomorphic. Useful for reducing byte payload for the crawling/scraping use case.
  • curl --header "prerender-remove-script-tags: true"
prerender-remove-trailing-slash
  • Removes trailing slash from URLs submitted to the API. e.g. example.com/docs/ -> example.com/docs. Originally intended for the pre-rendering use case, the primary purpose is to achieve a higher server cache hit rate. The SEO best practice is to use to 301s to redirect trailing slashes to non trailing slashes before service.headless-render-api.com is hit and/or using link rel canonical tag.
  • curl --header "prerender-remove-trailing-slash: true"
prerender-follow-redirects
  • By default, if your origin server returns 301/302, headless-render-api.com will just return that outright - which is appropriate for the common use case of proxying traffic through headless-render-api.com. If using the API in a crawling/scraping/batching you may want to follow redirects.
  • curl --header "Prerender-Follow-Redirects: true"

Optional HTTP Headers for Device Metrics

prerender-device-width
  • Overrides device screen width (default: 1366)
  • curl --header "prerender-device-width: 800"
prerender-device-height
  • Overrides device screen height (default: 768)
  • curl --header "prerender-device-height: 600"
prerender-device-is-mobile
  • Whether to emulate mobile device (default: false).

    This includes viewport meta tag, overlay scrollbars, text autosizing and more. In other words, whether the meta viewport tag is taken into account.

    an example of "viewport meta tag":
  • curl --header "prerender-device-is-mobile: true"

Optional HTTP Headers for Configure Server Response

prerender-with-screenshot
  • Changes the response from text/html to application/json and returns an object 2 fields: {body, screenshot}
    The values are base64 encoded:
    • body is the normal pre-rendered response
    • screenshot is a base64 encoded PNG

    This is useful for savings screenshots or injecting them as open graph images. Often used with Prerender-With-Metadata

  • curl --header "Prerender-With-Screenshot: true"
prerender-with-metadata
  • Changes the response from text/html to application/json and returns an object 3 fields: {body, meta, links}
    The values are base64 encoded:
    • body is the normal pre-rendered response
    • meta is an object that includes  { title, metaDescription, h1, ogImage, ogTitle, twitterCard }
    • links is an array of URLs/paths on the page

    This is useful for capturing SEO-relevant metadata during the pre-render.

  • curl --header "Prerender-With-Metadata: true"

Optional in-page configuration

Controlling Status Codes
  • The API endpoint returns whatever status the origin returns, which for single-page apps (index.html) is usually 200, unless you change it with these tags. This is irrelevant if you're just using service.headless-render-api.com as a batch crawling/scraping tool.
  • 404 status code informs a bot that the page should be removed from their index
    • <meta name="prerender-status-code" content="404">
  • 301 status code informs a bot that their is newer content elsewhere
    • <meta name="prerender-status-code" content="301">
    • <meta name="prerender-header" content="Location: http://www.example.com"> (note: no other headers, currently, can be modified like this)

HTTP Error Codes

Our official libraries are configured to propogate 400 errors to what a visitor of your website would see, but the libraries are (or should be) configured to hide rate limiting (429) and 5xx errors and fallback to non-prerendered content.

400
  • Invalid Request
  • There will be an error message in the HTML, fix your request and retry
  • Example causes:
    • malformed URL
    • a localhost URL/IP
    • or a page responds with content-type: application/octet-stream
429
  • Rate limited
  • Requests made without API tokens (or expired/missing billing information) will get this - see pricing
500
  • General Error
  • Example causes:
    • 10s (timeout) while waiting for a page to finish rendering (waits until all in-flight requests finish, load event, domContent event etc.)
    • or HTTPS Page is making HTTP (non secure) requests
    • ...or a random bug on our end
  • The error will show up in the headless-render-api.com web UI (after you sign in)
502
  • Bad Gateway (your origin returned 5xx)
  • It means we received a 5xx when trying to visit your page
  • Or it means we received a 403 (forbidden). Typically this means your page is behind a login wall or a firewall (like Cloudflare) is blocking the headless-render-api user-agent (make an exception to allow user-agents matching /prerendercloud/)
  • Probably not retryable. It depends on the site you're requesting. If it's your site, make sure it's up and running correctly
503
  • Over capacity (Rare)
  • Retry the request with some backoff
  • We'll see the error and our autoscaler will increase capacity within 5 minutes, but you should email us anyway: support@headless-render-api.com
504
  • Gateway Timeout (Rare)
  • Retry the request with some backoff
  • This is unexpected and should not happen. We'll see it, but you should email us anyway: support@headless-render-api.com