/oss · 1 merged

apify/crawlee

A toolkit programmers use to build bots that crawl websites and collect data from them.

the project on GitHub (opens in a new tab)
  1. 14 Oct 2026

    Sitemap pages wrapped in extra spacing are no longer lost

    What was broken
    A sitemap is a file listing a site's pages for crawlers. When a page address had spaces or line breaks around it in a quoted block, the tool kept the space instead, dropped the page, and could crawl nothing.
    What I changed
    The tool now ignores bits of text that are only spaces or line breaks.
    Why it matters
    Crawls that start from these sitemaps visit every page they list.

    see the code, #4250 (opens in a new tab)

← every project