Curl Library Installation

Question:
Hello, my console returns an error `curl/curl.h that states no such file or directory exists. Do I need to download the libraries as I do not have permission to access root (obv)? This is a school final project so I want to ensure I am do this correctly.

Repl link:
(https://replit.com/@nateASC4/WebCrawler)


#include <stdio.h> // scanning through files 
#include <string.h> // scanning through files 
#include <curl/curl.h> // to send HTTP requests 
#include <libxml/HTMLparser.h> // for parsing HTML and XML 


#include <pthread.h> ///for POSIX multithreading 
#include <stdlib.h> // scanning through files 


#define NUM_THREADS 6

#define MAX_LINKS 10
//max URL length
#define MAX_URL_LENGTH 2083 
#define buffer[MAX_URL_LENGTH]


//In this program we will follow SOLID principles to ensure clarity and improve functionality of our code

//scan a file for URL
      // Check if filename is provided as an argument
int step1_scanFile(){
  int scan = 0; 
    if (scan != 2) {
        printf("Usage: ./scan_urls <filename>\n");
        return 1;
    }

    // Open file
    FILE* fp = fopen("url.txt", "r");
    if (fp == NULL) {
        printf("Error: could not open file %s\n", "url.txt");
        return 1;
    }
  //we need to iterate through the list in a loop

    char url[MAX_URL_LENGTH];
    while (fgets(url, MAX_URL_LENGTH, fp) != NULL) {
        // Remove trailing newline character if present
        if (url[strlen(url) - 1] == '\n') {
            url[strlen(url) - 1] = '\0';
        }
      
        // Check if url starts with http:// or https://
        if (strncmp(url, "http://", 7) == 0 || strncmp(url, "https://", 8) == 0) { 
          //use strncmp to avoid buffer overflows
            printf("Found URL: %s\n", url);
        }
        //if url does not exist 
      else {
        printf("%s","404:URL not FOUND");
      }
    }
    // Close file
    fclose(fp);

    return 0;
}
//initialize the libcurl library 
void step0_initlibcurl(){
  CURL* curler;
  Curlcode res;

  
  res = curl_easy_perform(curl); 

      // Check if the curl request was successful
  if (res != CURLE_OK) {
      printf("Error performing curl request: %s\n", curl_easy_strerror(res));
      return 1;```

add it via Nix: in shell:

sed -i 's/];/  pkgs.curl.dev\n  &/' replit.nix
sed -i '4s/$/ -lcurl' Makefile
kill -1 `pidof ccls`
3 Likes

Thank you! The error for curl disappeared. Do you also happen to know how to download libxml into Nix. I am building a WebCrawler in C?

Try this: NixOS Search

Whenever you need to search for a package go to search.nixos.org

2 Likes
  1. Click Files ⋮ > Show hidden files
  2. Add to the replit.nix file, before the closing }:
  env = {
    C_INCLUDE_PATH = "${pkgs.libxml2.dev}/include/libxml2";
  };
  1. Run in shell:
{
sed -i '/^override C/s/$/ -lxml2/' Makefile
pkill ccls
}

How did I get the code for #2?

Do read this so you can hopefully add packages in future by yourself
First, I search for the package on search.nixos.org.
I add .dev at the end because this is what you do when

  • you want a header
  • and you see a dev output in the search result on search.nixos.org.

Adding to the deps as with curl doesn’t work: this library is packaged weirdly so the actual library that you have to include is within a subfolder of /include (/include/libxml2).
How did I know that?
I want to inspect the package’s contents. First I add to replit.nix:

env = {
  XML = "${pkgs.libxml2.dev}/include";
}

Our include starts with libxml/, so we want to be able to include a libxml folder, so in Shell, I run:

PROMPT_COMMAND='cd $XML
shopt -s globstar
echo **/libxml/
exit'

Output: libxml2/libxml. So, when I add libxml/ to the $C_INCLUDE_PATH, I must add /include/libxml2. I change the env section to:

    C_INCLUDE_PATH = "${pkgs.libxml2.dev}/include/libxml2";

When I want to add another library that’s packaged weirdly and can’t be added through deps, I have to add a : before the next path, e.g.

     C_INCLUDE_PATH = "${pkgs.libxml2.dev}/include/libxml2:${pkgs.SDL2.dev}/include/SDL2";

You can usually use builtins, string functions and with to simplify this when it becomes too long and repetitive. A good example.

3 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.