How can one work around restrictions imposed by websites like Wikipedia that block data extraction through PHP scripts?

We can work around restrictions imposed by websites like Wikipedia that block data extraction through PHP scripts by using web scraping techniques. One way to do this is by using cURL to make HTTP requests to the website and then parsing the HTML content to extract the desired data. 

<?php
$url = 'https://en.wikipedia.org/wiki/Main_Page';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
curl_close($ch);

$dom = new DOMDocument();
@$dom->loadHTML($output);

$xpath = new DOMXPath($dom);
$elements = $xpath->query('//h1[@id="firstHeading"]');
foreach ($elements as $element) {
    echo $element->nodeValue;
}
?>

Keywords

web scraping data extraction PHP cURL user agent

Related Questions