What is the significance of handling relative URLs in a web crawler script?

Handling relative URLs in a web crawler script is significant because it ensures that all URLs are correctly formatted and can be properly processed by the crawler. Without handling relative URLs, the crawler may encounter broken links or fail to navigate to the correct pages on a website. To solve this issue, we can use PHP's built-in functions to convert relative URLs to absolute URLs before processing them in the crawler. 

function resolveUrl($baseUrl, $url) {
    $urlParts = parse_url($url);
    if (isset($urlParts['scheme'])) {
        return $url;
    }
    
    $baseParts = parse_url($baseUrl);
    $scheme = $baseParts['scheme'];
    $host = $baseParts['host'];
    $path = $baseParts['path'];
    
    if (substr($url, 0, 1) == '/') {
        return "$scheme://$host$url";
    } else {
        $path = rtrim($path, '/') . '/';
        return "$scheme://$host$path$url";
    }
}

$baseUrl = 'https://example.com';
$url = '/page1.html';
$absoluteUrl = resolveUrl($baseUrl, $url);
echo $absoluteUrl;

Keywords

web crawler PHP script relative URLs link normalization URL parsing

Related Questions