Log In · Signup for Free
EN  DE  FR  ES  IT  Русский  한국어  日本語  简体中文  عربي
🤖
Live and expired websites.GUIDE

Datasets .GUIDE All Robots.txt

Download a current list of Datasets .GUIDE All Robots.txt with website, company, location and technology detection fields.

One-time dataset price
$495.00

One-time delivery in Parquet or TSV format.

Buy Dataset

Compare Team Plan Ultra

Dataset facts
Domains
9,458
Detections
430,973
Generated
2026-07-26 UTC
Formats
Parquet, TSV
Scope
.GUIDE
What's included
  • ✓ Domain and technology detection history
  • ✓ Company and website location signals
  • ✓ Spend, revenue and employee estimates
  • ✓ Traffic and authority rankings
  • ✓ Available business contact and social fields
View fields and technical schema →
Find another dataset
Browse all datasets
Dataset sampleParquet sampleTSV sample

Preview representative rows before purchasing. Full downloads contain all available fields and records.

DomainTechnologyFirst detectedLast detectedCompanyLocationSpend
thenice.guideGoogle2023-03-212024-10-14$122
thenice.guideGoogle Cloud2023-03-212024-10-14$122
thenice.guideGoogle Cloud Iowa2023-03-212024-09-21$122
thenice.guideSPF2023-03-212024-10-24$122
thethrive.guideSquarespace2024-09-172026-07-24$122
thethrive.guideLocalBusiness Schema2024-09-172026-07-24$122
thethrive.guideFacebook2024-09-172026-07-24$122
thethrive.guideGoogle Analytics 42024-09-172026-07-24$122
commodity.guideHetzner DNS2024-12-072026-07-25commodity seHerisau, CH$235
commodity.guideSPF2024-12-072026-07-25commodity seHerisau, CH$235
commodity.guideGerman Server Location2025-01-092026-07-25commodity seHerisau, CH$235
commodity.guideKey-Systems Registrar2024-11-232025-11-22commodity seHerisau, CH$235
discipleship.guideFacebook for Websites2020-06-022026-07-22Table Church$122
discipleship.guideViewport Meta2020-06-022026-07-22Table Church$122
discipleship.guideIPhone / Mobile Compatible2020-06-022026-07-22Table Church$122
discipleship.guideSquarespace2020-06-022026-07-22Table Church$122
atheist.guideTrellian2023-12-132024-04-12$1,900
atheist.guideSPF2023-12-132024-11-07$1,900
atheist.guideAustralian Server Location2024-01-032024-07-09$1,900
atheist.guideParkLogic DNS2024-04-272024-11-07$1,900
brianneelizabeth.guideGoogle Domains2025-04-082026-02-26$122
brianneelizabeth.guideGoogle DNS2025-04-082026-02-26$122
brianneelizabeth.guideGoogle Cloud DNS2025-04-082026-02-26$122
brianneelizabeth.guideSquarespace Hosted2025-04-082026-02-26$122
extraders.guideHSTS2024-05-212026-04-14$1,565
extraders.guideImperva Bot Detection and Mitigation2024-05-212025-09-15$1,565
extraders.guideChinese2024-05-212026-04-14$1,565
extraders.guideSSL by Default2024-05-212026-07-27$1,565
musa.guideRemix2024-07-152025-01-26Musa Guide Limited$166
musa.guideHTML5 History API2024-07-152026-07-26Musa Guide Limited$166
musa.guideViewport Meta2024-07-152026-07-26Musa Guide Limited$166
musa.guideJavaScript Modules2024-07-152026-07-26Musa Guide Limited$166
Related datasets
Live and Expired websites using Robots.txt128,929,443Live and Expired websites using Robots.txt in .COM TLD78,255,867Live websites using Robots.txt33,854,084Live websites using Robots.txt in .COM TLD21,637,530Live and Expired websites using Robots.txt in United States7,953,549Live and Expired websites using Robots.txt in .ORG TLD5,191,806Live and Expired websites using Robots.txt in .NET TLD5,187,818Live and Expired websites using Robots.txt in .RU TLD3,706,386