How do I handle paywalls and login walls in agentic web scraping?

Handle paywalls and login walls in agentic web scraping by respecting them in most cases — bypass tactics violate terms of service and create legal/ethical risk. The defensible patterns are: (1) use publisher APIs where available (NYT, FT, Bloomberg all have agent-accessible APIs); (2) authenticate as a paying user with stored session cookies passed into the fetch tool; (3) use Google cached / archive.org for occasional reads with attribution; (4) accept the paywall preview and let the agent reason about what it can see. AgentFetch supports session cookies — pass cookies={...} to fetch_url and it'll authenticate against the publisher as your account. For login walls on internal tools (your company's Confluence, Notion, internal dashboards), the right answer is usually a dedicated MCP server for that tool (mcp-notion, mcp-confluence) which uses proper API auth instead of HTML scraping. Bypassing paywalls via cookie tricks, archive lookups at scale, or paywall-stripper scripts is a fast path to getting the agent's IP banned and the developer sued. For agent applications, the cleaner architecture is "fetch tool for public web + auth-aware connectors for paid sources."