Normalize a URL
-
Updated
Mar 10, 2024 - JavaScript
#
url-normalization
Here are 13 public repositories matching this topic...
Extract and decompose URLs (including emails, which are conceptually a part of URLs) with robust patterns.
url-parsing url-parser email-parsing uri-template url-validation email-extractor email-parser url-extractor url-normalization url-normalizer
-
Updated
Jul 27, 2024 - JavaScript
Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
url crawler uri crawling tld recon url-parsing url-parser url-validation url-checker url-normalization
-
Updated
Dec 2, 2024 - Python
Remove clutter from URLs and return a canonicalized version
-
Updated
Jun 3, 2024 - Python
Get a stable, canonical version of any URL, with DNS and HTTPS checks, redirects, tracker stripping, and canonical link extraction!
javascript tracker tracking privacy amp npm-package compare-urls ssrf canonical-urls canonical url-normalization normalize-url sanitize-url
-
Updated
Dec 10, 2024 - JavaScript
🔗🧹 Normalize URLs to a standardized form. HTTPS by default, flexible configuration, custom protocols, domain extraction, humazing URL, and punycode support. Both CJS & ESM modules available.
url typescript npm-package punycode normalizer esm cjs normalization url-normalization url-normalizer
-
Updated
Nov 9, 2024 - TypeScript
Allows you to remove ad/tracking query params from a given URL in Scala
-
Updated
Nov 20, 2024 - Scala
URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical
url-normalization url-normalizer url-canonicalization url-standardizer url-standardization url-comparer
-
Updated
May 2, 2024 - C#
Natural Language Precessing related notebooks (Machine Learning)
-
Updated
Aug 20, 2021 - Jupyter Notebook
A simple plugin to rewrite multiple domains or subdomains to a single domain in the site's HTML output.
-
Updated
Jan 22, 2020 - PHP
http url normalization for web crawlers
-
Updated
Sep 20, 2023 - C++
🔗 Pathor is a PHP library for normalizing, analyzing, and comparing URLs.
-
Updated
Dec 12, 2024 - PHP
A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.
rate-limiting http-requests error-handling html-parsing data-collection text-processing web-crawling content-extraction yaml-configuration data-scraping python-crawler modular-design metadata-storage url-normalization pdf-text-extraction structured-data-storage concurrent-crawling data-extraction-pipeline data-preservation-and-recovery
-
Updated
Nov 18, 2024 - Python
Improve this page
Add a description, image, and links to the url-normalization topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the url-normalization topic, visit your repo's landing page and select "manage topics."