What are HTTP Headers?
HTTP Headers are key-value pairs sent between clients and servers in HTTP requests and responses. They provide essential metadata about communication parameters, authentication, content format, caching directives, and security policies. Headers enable web browsers, APIs, proxies, and servers to coordinate effectively, implementing features like content negotiation, session management, security policies, and performance optimization.
Headers operate invisibly to end users but fundamentally enable modern web functionality. When you browse a website, your browser sends dozens of headers describing what content types you accept, what language you prefer, whether you have authentication credentials, and various security preferences. The server responds with headers indicating content type, caching rules, security policies, and metadata about the response.
Common Request Headers
Request headers accompany HTTP requests from clients to servers, providing information about the client, requested resource, and how the server should process the request. The User-Agent header identifies the client software—browsers identify themselves as Chrome, Firefox, or Safari, while automated tools specify their names and versions. This identification helps servers optimize content delivery and detect automated access patterns.
The Accept header tells servers what content types the client can process. Browsers typically accept HTML, JSON, images, and various other formats. The Accept-Language header specifies preferred languages, allowing servers to return localized content. Accept-Encoding indicates compression algorithms the client supports, typically gzip, deflate, and Brotli, enabling bandwidth reduction through compression.
// Setting comprehensive request headers
const response = await fetch('https://api.example.com/data', {
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0',
'Accept': 'application/json, text/plain, */*',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Referer': 'https://example.com/',
'Origin': 'https://example.com'
}
});The Authorization header carries authentication credentials enabling access to protected resources. Bearer tokens, API keys, and Basic authentication credentials all transmit through this header. The Cookie header sends previously stored cookies back to servers, maintaining session state and user preferences across requests. Referer indicates the previous page URL, helping servers understand user navigation patterns and implement referrer-based access controls.
Response Headers
Response headers accompany HTTP responses from servers to clients, describing the response content, caching policies, security requirements, and server information. The Content-Type header specifies response format—application/json for JSON data, text/html for HTML pages, image/png for images. Clients use this to determine how to process response bodies correctly.
Cache-Control headers direct caching behavior for browsers, CDNs, and proxy servers. The “public” directive allows any cache to store responses. “Private” restricts caching to browsers only. “no-cache” requires revalidation before using cached copies. “max-age” specifies freshness duration in seconds. Proper cache headers dramatically improve performance by enabling content reuse without repeatedly fetching from origin servers.
// Reading and processing response headers
async function analyzeResponse(url: string) {
const response = await fetch(url);
// Extract key headers
const contentType = response.headers.get('content-type');
const cacheControl = response.headers.get('cache-control');
const rateLimit = response.headers.get('x-ratelimit-remaining');
const etag = response.headers.get('etag');
console.log('Content-Type:', contentType);
console.log('Cache-Control:', cacheControl);
console.log('Rate Limit:', rateLimit);
console.log('ETag:', etag);
return response.json();
}Set-Cookie headers instruct browsers to store cookies for subsequent requests. Location headers specify redirect targets for 301, 302, and other redirect responses. ETag headers provide resource version identifiers enabling conditional requests that reduce bandwidth. Server headers identify server software, though security-conscious deployments often omit or obfuscate this information to avoid revealing infrastructure details to potential attackers.
Authentication Headers
Authentication headers transmit credentials from clients to servers proving identity and authorization. Bearer token authentication dominates modern web APIs, sending JSON Web Tokens (JWT) or opaque tokens in the Authorization header. The format follows Bearer {token} where token contains the credential. APIs validate tokens on each request, checking signatures, expiration dates, and encoded permissions.
Basic authentication encodes username and password combinations in Base64 format transmitted as Basic {encoded_credentials}. While simple to implement, Basic authentication requires HTTPS protection since Base64 encoding provides no security—it merely encodes credentials without encryption. Many APIs consider Basic authentication acceptable only over encrypted connections or for development environments.
// Bearer token authentication
async function authenticatedRequest(url: string, token: string) {
const response = await fetch(url, {
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json'
}
});
if (response.status === 401) {
throw new Error('Authentication failed - invalid or expired token');
}
return response.json();
}
// Basic authentication
function createBasicAuth(username: string, password: string): string {
const credentials = btoa(`${username}:${password}`);
return `Basic ${credentials}`;
}
const response = await fetch(url, {
headers: {
'Authorization': createBasicAuth('user', 'password')
}
});API key authentication uses custom headers or query parameters to transmit keys. Common header names include X-API-Key, Authorization with “ApiKey” scheme, or service-specific headers like X-GitHub-Api-Version. API keys provide simpler authentication than tokens but offer less security features—they typically don’t expire automatically and can’t encode permissions directly.
Content Negotiation Headers
Content-Type headers specify request and response body formats enabling proper parsing and processing. JSON APIs use application/json. HTML pages use text/html. Form submissions use application/x-www-form-urlencoded or multipart/form-data depending on whether files are included. XML services use application/xml or text/xml. Binary data uses application/octet-stream.
Clients must set Content-Type correctly when sending data to servers. POST and PUT requests including JSON bodies require Content-Type: application/json headers. File uploads typically use multipart/form-data, allowing multiple files and form fields in single requests. URL-encoded form data uses application/x-www-form-urlencoded where form fields encode as key=value pairs separated by ampersands.
// JSON content type
await fetch('https://api.example.com/users', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
name: 'John Doe',
email: 'john@example.com'
})
});
// Form data with file upload
const formData = new FormData();
formData.append('name', 'John');
formData.append('avatar', fileBlob);
await fetch('https://api.example.com/upload', {
method: 'POST',
body: formData
// Don't set Content-Type - browser handles multipart/form-data automatically
});
// URL encoded form
const params = new URLSearchParams({
username: 'john',
password: 'secret'
});
await fetch('https://api.example.com/login', {
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
body: params.toString()
});Accept headers describe what content types clients prefer receiving. Browsers send comprehensive Accept headers supporting HTML, JSON, XML, images, and generic types. APIs typically request specific formats like application/json. Servers examine Accept headers and respond with appropriate content types, implementing content negotiation where single URLs serve different formats based on client preferences.
CORS Headers
Cross-Origin Resource Sharing (CORS) headers control whether browsers allow JavaScript from one origin to access resources from different origins. Without CORS headers, browsers block cross-origin requests protecting users from malicious scripts accessing sensitive data from other sites. CORS headers enable legitimate cross-origin access while maintaining security.
Origin headers indicate the source origin of requests. Browsers automatically include Origin headers on cross-origin requests specifying the protocol, domain, and port of the requesting page. Servers examine Origin headers deciding whether to allow access. Access-Control-Allow-Origin response headers specify which origins may access resources—either specific origins like “https://example.com” or the wildcard ”*” allowing any origin.
// Browser automatically sends CORS request headers:
// Origin: https://example.com
// Access-Control-Request-Method: POST
// Access-Control-Request-Headers: Content-Type, Authorization
// Server responds with CORS permission headers
export async function GET(request: Request) {
const origin = request.headers.get('origin');
return new Response(JSON.stringify({ data: 'value' }), {
headers: {
'Content-Type': 'application/json',
'Access-Control-Allow-Origin': origin || '*',
'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE',
'Access-Control-Allow-Headers': 'Content-Type, Authorization',
'Access-Control-Allow-Credentials': 'true',
'Access-Control-Max-Age': '3600'
}
});
}Preflight requests occur before complex cross-origin requests. Browsers send OPTIONS requests asking servers whether the actual request is permitted. Access-Control-Request-Method indicates the intended HTTP method. Access-Control-Request-Headers lists headers the actual request will include. Servers respond with Access-Control-Allow-* headers granting or denying permission. Access-Control-Max-Age specifies how long browsers may cache preflight responses avoiding redundant OPTIONS requests.
Caching Headers
Cache-Control headers direct caching behavior throughout the request/response chain including browsers, CDN edge servers, and proxy caches. “Public” allows any cache to store responses. “Private” restricts caching to browsers, preventing shared caches like CDNs from storing potentially sensitive responses. “No-cache” requires revalidation before using cached copies. “No-store” prohibits caching entirely for highly sensitive data.
Max-age directives specify freshness duration in seconds. “max-age=3600” indicates responses remain fresh for one hour. “s-maxage” applies specifically to shared caches like CDNs, often set longer than max-age to enable aggressive CDN caching while requiring browsers to revalidate more frequently. “Immutable” indicates resources never change, enabling indefinite caching without revalidation—typically used for versioned static assets.
// Conditional request with ETag
async function fetchWithConditionalRequest(url: string) {
// Initial request
const response = await fetch(url);
const etag = response.headers.get('etag');
const data = await response.text();
console.log('Initial fetch completed');
// Later request with If-None-Match
const response2 = await fetch(url, {
headers: {
'If-None-Match': etag || ''
}
});
if (response2.status === 304) {
console.log('Not modified - using cached data');
return data;
}
console.log('Modified - fetching new data');
return response2.text();
}ETag headers provide resource version identifiers enabling conditional requests. Servers generate ETags representing specific resource versions—typically content hashes or version numbers. Clients include If-None-Match headers with cached ETags on subsequent requests. Servers compare request ETags against current resource versions, responding with 304 Not Modified when ETags match, saving bandwidth by avoiding redundant data transfer.
Security Headers
Security headers protect users from various web attacks by instructing browsers to enforce security policies. Strict-Transport-Security (HSTS) forces browsers to use HTTPS for all future requests to domains, preventing SSL stripping attacks. X-Content-Type-Options: nosniff prevents browsers from MIME-sniffing responses, defending against attacks exploiting MIME type confusion.
X-Frame-Options controls whether pages may be embedded in frames, preventing clickjacking attacks. DENY prohibits all framing. SAMEORIGIN allows framing only by pages from the same origin. Content-Security-Policy (CSP) defines comprehensive security policies controlling script sources, image sources, styles, and other resource types. CSP represents the most powerful security header, effectively preventing XSS attacks when properly configured.
// Applying comprehensive security headers
export async function GET() {
return new Response('Secure content', {
headers: {
'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload',
'X-Content-Type-Options': 'nosniff',
'X-Frame-Options': 'SAMEORIGIN',
'X-XSS-Protection': '1; mode=block',
'Content-Security-Policy': "default-src 'self'; script-src 'self' 'unsafe-inline'",
'Referrer-Policy': 'strict-origin-when-cross-origin',
'Permissions-Policy': 'geolocation=(), microphone=(), camera=()'
}
});
}Referrer-Policy headers control what referrer information browsers include in requests. “strict-origin-when-cross-origin” sends full URLs for same-origin requests but only origins for cross-origin requests, balancing analytics needs with privacy protection. Permissions-Policy headers disable or control powerful browser features like geolocation, camera, and microphone access, reducing attack surface and protecting user privacy.
Custom Headers
Custom headers enable application-specific functionality beyond standard HTTP headers. Proxy services use custom headers for configuration—CorsProxy uses x-cors-api-key for authentication and x-cors-proxy-type for proxy type selection. Rate limiting headers like X-RateLimit-Remaining communicate API usage quotas to clients enabling intelligent request throttling.
API versioning headers specify which API version clients request. GitHub uses X-GitHub-Api-Version while other services embed versions in Accept headers using custom media types. Tracing headers like X-Request-ID uniquely identify requests enabling request tracking across distributed systems. Load balancers and proxies add X-Forwarded-For headers preserving original client IP addresses.
// Using custom headers with various services
const corsProxyRequest = await fetch(
`https://corsproxy.io/?url=${encodeURIComponent('https://api.example.com/data')}&key=your-api-key&type=residential&colo=fra`
);
// GitHub API with version header
const githubRequest = await fetch('https://api.github.com/user', {
headers: {
'Accept': 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'Authorization': 'Bearer github_token'
}
});
// Custom tracing header
const tracedRequest = await fetch('https://api.example.com/data', {
headers: {
'X-Request-ID': crypto.randomUUID(),
'X-Correlation-ID': 'user-action-123'
}
});Monitoring and debugging headers help developers understand request processing. X-Response-Time indicates server processing duration. X-Served-By identifies which server or CDN node handled requests. X-Cache shows whether responses served from cache (HIT) or origin servers (MISS). These headers provide valuable insights during development and troubleshooting without affecting functionality.
Headers for Web Scraping
Web scraping requires realistic header combinations mimicking genuine browser requests. Sites implementing bot detection examine headers identifying automated access patterns. Missing User-Agent headers, unusual Accept header values, or absent Sec-Fetch-* headers immediately flag requests as potentially automated.
Realistic User-Agent strings should match actual browsers including version numbers and platform details. Accept headers must include appropriate MIME types in correct order with quality values. Accept-Language headers should specify reasonable language preferences. Accept-Encoding must list compression algorithms browsers actually support. Connection headers should specify “keep-alive” matching browser behavior.
// Realistic browser headers for scraping
async function scrapeWithRealisticHeaders(url: string) {
const headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Referer': 'https://www.google.com/',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'none',
'Sec-Fetch-User': '?1',
'Cache-Control': 'max-age=0'
};
return fetch(url, { headers });
}Sec-Fetch-* headers provide additional context about requests. Sec-Fetch-Dest indicates request destination type (document, image, script). Sec-Fetch-Mode specifies request mode (navigate, cors, no-cors). Sec-Fetch-Site describes relationship between requester and destination (same-origin, cross-site, none). Modern browsers automatically include these headers, and their presence with correct values helps requests appear legitimate.
Using HTTP Headers with CorsProxy
CorsProxy simplifies header management for proxy-routed requests by handling complex proxy authentication and configuration through URL parameters instead of headers. This approach eliminates the complexity of proxy header management while maintaining full control over request headers forwarded to destination servers.
When using CorsProxy, standard HTTP headers like User-Agent, Accept, Authorization, and custom application headers pass through transparently to destination servers. CorsProxy automatically adds appropriate proxy headers, handles CORS headers, and ensures responses include correct metadata. This separation of concerns means applications focus on business logic headers while CorsProxy manages proxy infrastructure headers.
// CorsProxy with custom headers forwarded to destination
const response = await fetch(
`https://corsproxy.io/?url=${encodeURIComponent('https://api.example.com/data')}&key=your-api-key&type=residential&colo=fra`,
{
headers: {
// These headers forward to destination server
'Authorization': 'Bearer destination-api-token',
'User-Agent': 'MyApp/1.0',
'Accept': 'application/json',
'X-Custom-Header': 'custom-value'
}
}
);
// CorsProxy handles CORS automatically
const data = await response.json();Rate limiting headers from destination APIs pass through CorsProxy responses unchanged, allowing applications to implement intelligent throttling based on actual API limits. Custom tracking headers, correlation IDs, and debugging headers work normally through CorsProxy. Security headers from destinations apply correctly, maintaining end-to-end security while benefiting from proxy infrastructure for performance and reliability.
Best Practices
Set Content-Type headers appropriately for all requests with bodies. POST and PUT requests should specify whether they’re sending JSON, form data, or other formats. Servers reject requests with missing or incorrect Content-Type headers, or worse, misinterpret request bodies causing data corruption or security vulnerabilities.
Implement proper authentication using secure methods appropriate for your use case. Bearer tokens suit most modern APIs. Basic authentication requires HTTPS encryption. API keys should transmit through headers rather than query parameters avoiding exposure in server logs and browser history. Rotate credentials regularly and never commit them to version control systems.
Use caching headers aggressively for static content and carefully for dynamic data. Static assets like images, CSS, and JavaScript files benefit from long cache durations reducing bandwidth and improving performance. Dynamic content requires shorter cache durations or revalidation strategies ensuring users receive current data. Never cache sensitive user-specific information that could leak between users.
