PHP Basics: Accessing Remote URLs using cURL

In this post, I’m going back to basics to talk about accessing remote URLs with cURL.

In PHP, there are actually four ways to access a remote URL – fopen() fsockopen(), cURL extensions and HTTP_Request class from the PEAR library. Now, choosing one way over another really depends on your needs for simplicity, control, and portability.

Generally, I prefer using cURL simply because it’s easy to understand and it’s really powerful. cURL supports a number of different protocols and gives your access to the response headers. It even has the capability to do parallel access if you call the ‘curl_multi’ group of functions (I’ll save this for another post one day).

To use cURL, you need to make sure that your PHP install has the library enabled. Read this posting on how to enable cURL for your PHP.

Fetching a URL with the GET Method
The simplest way to start using cURL is to fetch a URL with the GET method.

1
2
3
4
5
6

<?php
$c = curl_init('http://www.askaboutphp.com/robots.txt');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$page = curl_exec($c);
curl_close($c);
?>

So what is going on? The simple example above really maps out 4 stages of using cURL. You start by calling the curl_init() to get a handle on the URL you wan to access. Then, you set some options using curl_setopt().After that, you execute what you want to do with curl_exec(). Lastly, you close the handle using curl_close().

The control of the curl functions is all to do with what options you set using curl_setopt(). PHP lists out all the options available at curl_setopt() page. Frankly, this is probably the hardest part about using the cURL library – understanding what all these options do. It was pretty daunting when I first started using.

Following on, here are some of the more commonly used (at least by me anyway) options for cURL.

Fetching a protected URL with the GET Method
To do this, you need to set the CURLOPT_USERPWD option, like so:

1
2
3
4
5
6
7

<?php
$c = curl_init('http://www.askaboutphp.com/protected-page.html');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_USERPWD, 'askme:anything'); 
$page = curl_exec($c);
curl_close($c);
?>

Submitting data to a URL with the POST Method
In this case, you need to set the CURLOPT_POST and CURLOPT_POSTFIELDS option.

1
2
3
4
5
6
7
8

<?php
$c = curl_init('http://www.askaboutphp.com/submit.php');
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS, 'father=dad&mother=mum');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($c);
curl_close($c);
?>

The parameters you want to pass should be formatted like a querystring. If you are expecting some response after posting, then you need to enable the CURLOPT_RETURNTRANSFER option.

Fetching a URL with Custom Headers
You can also retrieve URLs that requires specific headers to be sent with the request. You can do so with CURLOPT_HTTPHEADER option.

1
2
3
4
5
6
7

<?php
$c = curl_init('http://www.askaboutphp.com/custom-header.php');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_HTTPHEADER, array('My-Custom-Header: PHPISCOOL'));
$page = curl_exec($c);
curl_close($c);
?>

cURL has special options for setting the Referer and User-Agent request headers —
CURLOPT_REFERER and CURLOPT_USERAGENT:

1
2
3
4
5
6
7
8
9

<?php
$c = curl_init('http://www.askaboutphp.com/fetch.php');
curl_setopt($c, CURLOPT_VERBOSE, 1);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_REFERER, 'http://www.askaboutphp.com/form.php');
curl_setopt($c, CURLOPT_USERAGENT, 'CURL via PHP');
$page = curl_exec($c);
curl_close($c);
?>

I’m going to stop here. If you’re new to cURL, hopefully this has been a good broad tutorial to get you started. In the Web 2.0 age, online services constantly pop-up to allow you to programmatically connect to them, cURL is the best tool you can have to help you access those services.

• Categories

  • Book Review (5)
  • News (12)
  • Pentaho (1)
  • PHP Tutorials (39)
  • Website Building/SEO (11)

• Tags

  Apache Beginners' Guide blogs books cacti cakephp charts codeigniter comment conference crimson editor errors examples filezilla getting started global warming google google analytics google charts Guides hosting PHP How-to introduction jquery Microsoft monitoring mysql nofollow open source osspac PHP PHP 101 php basics phpMyAdmin reviews rrdtool sample codes SEO source code tools tutorial variables webmasters website XAMPP Lite

• Recent Comments

  • Cacti – set up custom monitor script | motrogan.ro on Cacti: Using Cacti to monitor web page loading – Part 1
  • Best Codeigniter video tutorial and tuts for beginners - PHPnx on Codeigniter: Helpers, Plugins and Libraries
  • 【PHP】アナリティクスAPIを使った人気記事ランキング on Google Analytics API class for PHP
  • Arindam Barman on PHP and jQuery: Submit a form without refreshing the page
  • kalai on Codeigniter: Creating dynamic graphs using JQuery and FusionCharts

• Archives

  • 2010  2009  2008  2007

• Recent Posts

  • Codeigniter: Creating dynamic graphs using JQuery and FusionCharts
  • PHP and jQuery: Submit a form without refreshing the page
  • Google Analytics officially uses the Asynchronous Tracking tag
  • PHP Basics: Sorting Arrays
  • Google Analytics Master Class is back in Singapore!