Search results
1030 packages found
Crawler (spider) of site web pages by domain name
CrawleeOne is a framework built on top of Crawlee and Apify for writing robust and highly configurable web scrapers
A simple email extractor for obfuscated emails.
- email-address
- extractor
- obfuscated
- obfuscated-email
- obfuscated-emails
- parse
- regex
- obfuscator
- obfuscation
- harvester
- scraper
- crawler
- View more
xvideos.com api implementation.
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
A robust GitHub API crawler that walks a queue of GitHub entities retrieving and storing their contents.
An `URL` parser for crawling purpose.
- crawler-url-parser
- url-parser
- extract-url
- url-parse
- is-parent-url
- is-child-url
- url
- parser
- parse
- crawler
- extract
- extractor
- absolute
- relative
- View more
Yet another node torrent scraper based on x-ray. (Support iptorrents, torrentleech, torrent9, Yyggtorrent, ThePiratebay, torrentz2, 1337x, KickassTorrent, Rarbg, TorrentProject, Yts, Limetorrents, Eztv)
- torrent
- scraper
- crawler
- node
- api
- yggtorrent
- torrentz2
- torrentleech
- iptorrents
- 1337x
- thepiratebay
- tpb
- kickasstorrents
- rarbg
- View more
A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.
A package for detecting keywords on the Internet Computer
A JavaScript library that allows for the quick transformation of DOM documents into useful formats.
Find an element that matches a particular CSS selector on a website
Simple WAF to integrate with Node.js web systems
- waf
- nodejs
- firewall
- blocker
- filtering
- bot
- spider
- robot
- crawler
- useragent
- user-agent
- detector
- detect
- detection
- View more
nodejs lightweight scraper/crawler
Verify that a request is from Google crawlers using Google's DNS verification steps
Verify that a request is from Bing crawlers using Bing's DNS verification steps
Verify that a request is from Apple crawlers using DNS verification steps
Verify that a request is from Twitter crawlers using DNS verification steps