ScaleUpPulseBot
ScaleUpPulseBot is the crawler behind the morning news in the ScaleUp app. It reads articles about product decisions that publications put on the open web. The app shows each one as a short summary, with the publication's name and a link to the article.
How to recognise it
Every request carries this User-Agent:
ScaleUpPulseBot/1.0 (+https://scaleupapp.club/pulse-bot)
Every request also carries a From header with an address where a person can be reached. The crawler makes no request without one. It sends no cookies and no referrer, and it adds nothing to your web addresses. It does not run scripts.
What it fetches
- Your robots.txt.
- Your RSS or Atom feed. If the feed lists no articles, your home page.
- The articles listed there. An article it has already seen is not fetched again.
It reads pages and feeds only. A picture, a video or a file is never read, even when a feed links to one. It stops reading a page at 2 MB, gives up after 20 seconds, and follows at most 3 redirects, checking every rule on this page again at each one.
It never works around a paywall. A page that turns out to be behind one is not used, and its text is not kept.
When it considers a publication for the first time, it fetches the feed or the home page to see whether dated articles are listed there.
How often it comes
Twice a day, at about 04:30 and 13:00 India time. It waits at least 2 seconds between two requests to the same site.
robots.txt
ScaleUpPulseBot obeys robots.txt. It never fetches a path your rules disallow.
- It reads the group named ScaleUpPulseBot. If there is none, it reads the group for *.
- It reads Allow and Disallow. The longest rule that matches wins.
- A rule is matched from the start of the path. The patterns * and $ inside a rule are not read, so write each rule as the start of a path.
- It obeys Crawl-delay, up to 60 seconds.
- It reads your robots.txt again every 24 hours, so a change can take that long to reach it.
- If your robots.txt cannot be fetched, it reads that as no rules.
A page that carries noai in its X-Robots-Tag header or in its robots meta tag is not used, and its text is not kept.
What it keeps
- The readable text of the article, up to 12,000 characters.
- The article's web address, its title and the date it was published.
- The web address of the picture the page offers for link sharing, with your publication's name as the credit. The app loads the picture from that address. We never copy it.
It never keeps the page's HTML, pictures, sound or video.
The text of an article the app never showed is removed after 90 days. The text of an article the app did show is kept, as the record its summary was checked against.
What the app shows
A headline, a summary of 60 words at most, your publication's name and a link to the article. AI models write the summary and check it against the article's text.
How to be left out
Either of these works.
In robots.txt
User-agent: ScaleUpPulseBot Disallow: /
By email
We add your domain to a list the crawler checks before every request. From then on it makes no request to that domain or to any part of it, not even for robots.txt.
To have a summary that is already in the app taken down, write to the same address with the link.