Shell Checker

Why Scan Multiple URLs for Content?

You have a list of URLs. Maybe hundreds, maybe thousands. You need to check each one for specific content—a keyword, a pattern, an indicator that tells you something important about that page. Opening them one by one isn't realistic.

This tool automates that process. You define what you're looking for, paste your URLs, and let concurrent scanning do the work. Results come back with match status, response times, and any errors encountered. What would take hours of manual work finishes in minutes.

How to Scan Your URLs

The scanning process is straightforward:

1. Paste your URL list (one per line) or upload a text file
2. Enter the content pattern you're searching for
3. Set concurrency level based on your needs
4. Click scan and monitor real-time progress
5. Export results to CSV when complete

Configuring Your Search

Content Patterns

You're searching raw HTML source, not rendered content. This means:

• Text visible on the page will appear in the source
• HTML tags, attributes, and comments are searchable
• JavaScript code embedded in the page is included
• Dynamically loaded content (via AJAX) won't appear

Be specific with your patterns. Searching for "error" matches any page mentioning that word. Searching for "database connection error" or a specific error code narrows results significantly.

Concurrency Settings

Higher concurrency means faster scanning but comes with trade-offs:

• Low (1-5): Slow but gentle on target servers, rarely triggers blocks
• Medium (10-20): Good balance of speed and reliability for most use cases
• High (50+): Fast but may trigger rate limiting or IP blocks

The right setting depends on your targets. Scanning your own servers? Go high. Scanning external sites? Stay moderate.

Practical Use Cases

Content Monitoring

You manage multiple websites or landing pages. You need to verify specific content exists—a disclaimer, a tracking pixel, a legal notice. Rather than manually checking each page, scan your entire URL list and identify pages missing the required content.

Link Validation

You have a list of URLs from backlinks, directories, or partner sites. You want to verify your content or link still appears on each page. The scanner checks for your specific anchor text or URL pattern across all targets.

Research and Analysis

You're analyzing a set of websites for specific technologies, frameworks, or content patterns. Rather than visiting each manually, define patterns like "wp-content" (WordPress), "shopify" (Shopify stores), or specific meta tags that indicate what you're looking for.

Quality Assurance

After a site migration or update, you need to verify pages contain expected content. Scan all URLs with patterns that should appear—navigation elements, footer content, required scripts—and quickly identify any pages that failed to update properly.

Understanding Results

Each scanned URL returns several data points:

• Match status: Whether your pattern was found in the page source
• Response time: How long the server took to respond (in milliseconds)
• HTTP status: The response code (200, 404, 500, etc.)
• Error details: Any connection or parsing errors encountered

Response times help identify slow servers. HTTP status codes reveal broken links (404), server errors (500), or redirects (301/302). Error details explain why some URLs couldn't be scanned.

Working with Large URL Lists

For lists with thousands of URLs, consider these approaches:

• Pre-filter your list: Use Remove Duplicate Lines first to avoid scanning the same URL twice
• Clean your URLs: Use List Filter to remove obviously invalid entries
• Batch processing: For very large lists, process in batches of 1,000-5,000 URLs
• Time your scans: Run during off-peak hours if scanning sites that might rate-limit

Handling Common Issues

Rate Limiting

Some servers block rapid requests from a single source. If you see many timeouts or 429 (Too Many Requests) errors, reduce concurrency and add delays between requests. Some sites require scanning over longer periods.

SSL/TLS Errors

Certificate errors indicate HTTPS issues on the target. The site might have an expired certificate, misconfigured SSL, or be using self-signed certificates. These URLs typically need manual verification.

Timeout Errors

Timeouts happen when servers don't respond within the expected window. Causes include slow servers, network issues, or servers that drop connections from scanners. Consider increasing timeout settings or retrying failed URLs separately.

Privacy and Responsible Use

A few important notes about using this tool:

• Your URL list stays in your browser—we don't see or store what you scan
• Requests go directly from your browser to target servers
• Respect robots.txt and terms of service for sites you scan
• High-volume scanning can impact target servers—be considerate
• Use responsibly for legitimate purposes only