PHP Basics: Accessing Remote URLs using cURL

In this post, I’m going back to basics to talk about accessing remote URLs with cURL.

In PHP, there are actually four ways to access a remote URL – fopen() fsockopen(), cURL extensions and HTTP_Request class from the PEAR library. Now, choosing one way over another really depends on your needs for simplicity, control, and portability.

Generally, I prefer using cURL simply because it’s easy to understand and it’s really powerful. cURL supports a number of different protocols and gives your access to the response headers. It even has the capability to do parallel access if you call the ‘curl_multi’ group of functions (I’ll save this for another post one day).

To use cURL, you need to make sure that your PHP install has the library enabled. Read this posting on how to enable cURL for your PHP.

Fetching a URL with the GET Method
The simplest way to start using cURL is to fetch a URL with the GET method.

1
2
3
4
5
6
<?php
$c = curl_init('http://www.askaboutphp.com/robots.txt');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$page = curl_exec($c);
curl_close($c);
?>

So what is going on? The simple example above really maps out 4 stages of using cURL. You start by calling the curl_init() to get a handle on the URL you wan to access. Then, you set some options using curl_setopt().After that, you execute what you want to do with curl_exec(). Lastly, you close the handle using curl_close().

The control of the curl functions is all to do with what options you set using curl_setopt(). PHP lists out all the options available at curl_setopt() page. Frankly, this is probably the hardest part about using the cURL library – understanding what all these options do. It was pretty daunting when I first started using.

Following on, here are some of the more commonly used (at least by me anyway) options for cURL.

Fetching a protected URL with the GET Method
To do this, you need to set the CURLOPT_USERPWD option, like so:

1
2
3
4
5
6
7
<?php
$c = curl_init('http://www.askaboutphp.com/protected-page.html');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_USERPWD, 'askme:anything'); 
$page = curl_exec($c);
curl_close($c);
?>

Submitting data to a URL with the POST Method
In this case, you need to set the CURLOPT_POST and CURLOPT_POSTFIELDS option.

1
2
3
4
5
6
7
8
<?php
$c = curl_init('http://www.askaboutphp.com/submit.php');
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS, 'father=dad&mother=mum');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($c);
curl_close($c);
?>

The parameters you want to pass should be formatted like a querystring. If you are expecting some response after posting, then you need to enable the CURLOPT_RETURNTRANSFER option.

Fetching a URL with Custom Headers
You can also retrieve URLs that requires specific headers to be sent with the request. You can do so with CURLOPT_HTTPHEADER option.

1
2
3
4
5
6
7
<?php
$c = curl_init('http://www.askaboutphp.com/custom-header.php');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_HTTPHEADER, array('My-Custom-Header: PHPISCOOL'));
$page = curl_exec($c);
curl_close($c);
?>

cURL has special options for setting the Referer and User-Agent request headers —
CURLOPT_REFERER and CURLOPT_USERAGENT:

1
2
3
4
5
6
7
8
9
<?php
$c = curl_init('http://www.askaboutphp.com/fetch.php');
curl_setopt($c, CURLOPT_VERBOSE, 1);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_REFERER, 'http://www.askaboutphp.com/form.php');
curl_setopt($c, CURLOPT_USERAGENT, 'CURL via PHP');
$page = curl_exec($c);
curl_close($c);
?>

I’m going to stop here. If you’re new to cURL, hopefully this has been a good broad tutorial to get you started. In the Web 2.0 age, online services constantly pop-up to allow you to programmatically connect to them, cURL is the best tool you can have to help you access those services.

Share and Enjoy:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google Bookmarks
  • DZone
  • Propeller
  • Reddit
  • StumbleUpon
  • Technorati
  • Yahoo! Buzz
Posted on June 19, 2009 at 4:40 pm by Eldee · Permalink
In: PHP Tutorials · Tagged with: , ,

9 Responses

Subscribe to comments via RSS

  1. [...] this new post to the Ask About PHP blog they take a look at one of the basics of using PHP – fetching the data [...]

  2. Written by dumbo on June 23, 2009 at 6:23 am
    Permalink

    Nice and beginner friendly introduction!
    THX!

  3. Written by Binny V A on June 25, 2009 at 2:55 am
    Permalink

    I have written a wrapper for curl – it makes the job much easier – Load function

  4. [...] this new post to the Ask About PHP blog they take a look at one of the basics of using PHP – fetching the data [...]

  5. Written by Tom F on August 11, 2009 at 10:53 am
    Permalink

    Thank you! As a one time MS asp.net tinkerer (now reformed), this is just the type of introduction I need to kick-start development in a platform you can trust.

  6. Written by Martin on December 2, 2009 at 8:25 pm
    Permalink

    Great howto, Eldee! Thanks a lot! This is exactly what I was looking for.

  7. Written by Ivan on September 30, 2010 at 12:14 pm
    Permalink

    What a nice article Eldee,
    i have write about things you can do with cURL that maybe interest you:
    http://www.ivankristianto.com/web-development/programming/things-you-can-do-with-php-curl/1454/

  8. [...] a temat PHP = http://www.askaboutphp.com/tag/php, zastosowanie curl, jeszcze inny tutorial z smashing [...]

  9. Written by Sudhir on July 22, 2011 at 2:36 pm
    Permalink

    I am using cURL and it works for URL mapping to same webserver but returns nothing from remote URLs. What do I need to do to fix this? Has remote access being blocked by my provider?

Subscribe to comments via RSS

Leave a Reply