What is cURL?
cURL (Client URL) is a versatile command-line tool and library (libcurl) for transferring data using various network protocols. First released in 1997, cURL has become an essential tool for developers, system administrators, and DevOps engineers worldwide. The tool supports dozens of protocols including HTTP, HTTPS, FTP, SFTP, SCP, SMTP, and many others, making it invaluable for testing APIs, automating data transfers, debugging network issues, and scripting web interactions.
cURL operates entirely from the command line, requiring no graphical interface. This design makes it perfect for automation, remote server administration, continuous integration pipelines, and scenarios where GUI tools aren’t available. Developers use cURL to test API endpoints during development, verify HTTP header configurations, debug authentication issues, download files, upload data, and simulate various client behaviors that browsers or applications might exhibit.
Understanding cURL Basics
At its core, cURL sends HTTP requests to servers and displays the responses. The simplest cURL command fetches a URL and prints the response body to the terminal. While browsers handle HTML rendering, JavaScript execution, and complex user interactions, cURL provides raw access to HTTP communication, showing exactly what servers send without browser interpretation or modification.
cURL’s power lies in its flexibility. Unlike graphical tools that limit configuration options, cURL exposes every aspect of HTTP communication through command-line flags. You can control request methods, customize headers, manage cookies, configure authentication, set timeouts, follow redirects, handle SSL certificates, and implement virtually any HTTP interaction pattern. This granular control makes cURL indispensable for testing edge cases, replicating specific client behaviors, and automating complex request sequences.
// Equivalent cURL commands and fetch API
// cURL: curl https://api.example.com/users
const users = await fetch('https://api.example.com/users');
const data = await users.json();
// cURL: curl -H "Authorization: Bearer token123" https://api.example.com/users
const authenticatedUsers = await fetch('https://api.example.com/users', {
headers: {
'Authorization': 'Bearer token123'
}
});
// cURL: curl -X POST -H "Content-Type: application/json" -d '{"name":"John"}' https://api.example.com/users
const createUser = await fetch('https://api.example.com/users', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({ name: 'John' })
});HTTP Methods and Request Types
cURL supports all standard HTTP methods through the -X or —request flag. GET requests retrieve data from servers and represent the default method when no explicit method is specified. POST requests submit data to servers, typically creating new resources or triggering actions. PUT requests update existing resources by replacing them entirely. PATCH requests partially update resources by modifying specific fields. DELETE requests remove resources from servers.
The -d or —data flag specifies request body data for POST and PUT requests. When using -d, cURL automatically sets the Content-Type header to application/x-www-form-urlencoded unless explicitly overridden. JSON APIs require manually setting Content-Type to application/json using the -H flag. For file uploads, the -F or —form flag enables multipart/form-data encoding, automatically handling file boundaries and proper Content-Type configuration.
// Script to test API endpoints using cURL patterns
async function testAPIEndpoints() {
const baseUrl = 'https://corsproxy.io/?url=${encodeURIComponent("https://api.example.com")}&key=your-api-key&type=residential&colo=fra';
// GET request
const getResponse = await fetch(`${baseUrl}/users`);
console.log('GET status:', getResponse.status);
// POST request with JSON
const postResponse = await fetch(`${baseUrl}/users`, {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
name: 'John Doe',
email: 'john@example.com'
})
});
console.log('POST status:', postResponse.status);
// PUT request to update
const putResponse = await fetch(`${baseUrl}/users/123`, {
method: 'PUT',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
name: 'Updated Name',
email: 'newemail@example.com'
})
});
// DELETE request
const deleteResponse = await fetch(`${baseUrl}/users/123`, {
method: 'DELETE',
headers: {
'Authorization': 'Bearer token123'
}
});
return {
get: getResponse.status,
post: postResponse.status,
put: putResponse.status,
delete: deleteResponse.status
};
}Headers and Authentication
HTTP headers provide metadata about requests and responses, controlling content negotiation, caching, authentication, and various protocol behaviors. cURL’s -H or —header flag adds custom headers to requests. Multiple -H flags accumulate headers rather than overriding them, allowing comprehensive header configuration. Common headers include User-Agent identifying client software, Accept specifying desired response formats, Content-Type indicating request body format, and Authorization carrying authentication credentials.
Authentication represents a critical use case for cURL. Bearer token authentication dominates modern APIs, sending tokens in Authorization headers using the format Bearer {token}. Basic authentication encodes username and password combinations in Base64 format, suitable for testing but requiring HTTPS encryption for security. API keys transmit through custom headers like X-API-Key or via query parameters, though headers provide better security by avoiding log exposure.
The -u or —user flag simplifies Basic authentication, automatically encoding credentials and setting the Authorization header. For complex authentication flows requiring cookies or session tokens, cURL’s -c flag saves cookies to files and -b loads them for subsequent requests, enabling session management across multiple cURL invocations.
Using cURL with CorsProxy
CorsProxy enables browser-based applications to bypass CORS restrictions when accessing third-party APIs. While browsers enforce same-origin policies preventing direct cross-origin requests, cURL operates outside browser security contexts and doesn’t face CORS limitations. However, CorsProxy remains valuable for cURL users accessing rate-limited APIs, requiring IP rotation, or benefiting from geographic distribution.
When using CorsProxy with cURL, the URL parameter specifies the destination while additional parameters control proxy behavior. The key parameter authenticates requests, type selects proxy type (residential, datacenter, mobile), and colo chooses geographic location. CorsProxy handles routing, IP rotation, and header management transparently, allowing cURL commands to focus on application logic rather than proxy infrastructure.
// TypeScript wrapper for cURL-like requests through CorsProxy
class CurlProxyClient {
constructor(
private apiKey: string,
private proxyType: 'residential' | 'datacenter' | 'mobile' = 'residential',
private colo: string = 'fra'
) {}
async request(url: string, options: RequestInit = {}) {
const proxyUrl = `https://corsproxy.io/?url=${encodeURIComponent(url)}&key=${this.apiKey}&type=${this.proxyType}&colo=${this.colo}`;
const response = await fetch(proxyUrl, {
...options,
headers: {
...options.headers,
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
});
return response;
}
async get(url: string, headers: Record<string, string> = {}) {
return this.request(url, { headers });
}
async post(url: string, data: any, headers: Record<string, string> = {}) {
return this.request(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
...headers
},
body: JSON.stringify(data)
});
}
async download(url: string, outputPath: string) {
const response = await this.get(url);
const blob = await response.blob();
// In Node.js environment, write to file
// In browser, create download link
return blob;
}
}
// Usage
const client = new CurlProxyClient('your-api-key', 'residential', 'fra');
// GET request
const users = await client.get('https://api.example.com/users');
const usersData = await users.json();
// POST request
const newUser = await client.post('https://api.example.com/users', {
name: 'John Doe',
email: 'john@example.com'
});Advanced Features and Output Control
cURL provides extensive control over output formatting and request behavior. The -i or —include flag displays response headers alongside the body, useful for debugging header-related issues or verifying CORS configurations. The -I or —head flag sends HEAD requests retrieving only headers without response bodies, perfect for checking resource availability or metadata without downloading content.
The -v or —verbose flag enables detailed logging showing complete request/response communication including SSL handshake details, connection information, and protocol negotiations. This verbose output proves invaluable when debugging SSL certificate issues, proxy configurations, or redirect chains. Conversely, -s or —silent suppresses progress meters and error messages, ideal for scripting where output cleanliness matters.
The -o flag writes responses to specified files instead of standard output, while -O uses the URL’s filename. The -L or —location flag follows HTTP redirects automatically, essential for APIs using redirects for load balancing or canonicalization. The -w or —write-out flag formats custom output using variables like http_code, time_total, and size_download, enabling performance monitoring and response analysis.
Best Practices and Common Patterns
Always include appropriate User-Agent headers when scraping websites or accessing APIs that track client software. Many services rate-limit or block requests with missing or suspicious User-Agent values. Set realistic User-Agent strings matching browsers or identifying your bot appropriately to avoid being flagged as malicious automated traffic.
Handle authentication securely by avoiding command-line credential exposure. Use environment variables or credential files instead of embedding sensitive tokens directly in cURL commands. Command-line arguments often appear in process listings, shell history files, and server logs, potentially exposing credentials to unauthorized users.
Implement proper error handling in cURL scripts. Check HTTP status codes using the -w flag to detect failures, use -f or —fail to make cURL exit with error codes on HTTP errors, and combine with shell scripting conditional logic for robust automation. Always set reasonable timeouts using —connect-timeout and —max-time preventing indefinite hangs on network issues or unresponsive servers.
