What are some common PHP functions or libraries that can be used to crawl websites for links and store them in a MySQL database?

When crawling websites for links and storing them in a MySQL database, you can use PHP functions like `file_get_contents()` to fetch the website's content, `preg_match_all()` to extract links from the content, and MySQL functions to store the links in a database. You can create a script that loops through a list of URLs, crawls each website, extracts links, and inserts them into a database table. 

<?php

// Connect to MySQL database
$servername = "localhost";
$username = "username";
$password = "password";
$dbname = "database";
$conn = new mysqli($servername, $username, $password, $dbname);

// List of URLs to crawl
$urls = array("http://example.com", "http://example2.com");

foreach ($urls as $url) {
    // Fetch website content
    $content = file_get_contents($url);

    // Extract links using regular expression
    preg_match_all('/<a\s[^>]*href=([\"\']??)([^\" >]*?)\\1[^>]*>(.*)<\/a>/siU', $content, $matches);

    // Insert links into database
    foreach ($matches[2] as $link) {
        $sql = "INSERT INTO links (url) VALUES ('$link')";
        $conn->query($sql);
    }
}

// Close database connection
$conn->close();

?>

Keywords

PHP web crawling link extraction MySQL database data storage

Related Questions