Skip to the content
New on Record

For site owners: public-records-check

If public-records-check is in your server logs, this page explains it.

What it is

public-records-check is the automated checker behind New on Record, a directory of where Ohio's governments post their public records.

Once a night, it opens the page or file where your site posts its newest records, such as bids, agendas or permits. It checks that the page still opens and still shows records.

It is not a search engine. It is not an AI search or training program. Some robots.txt files now name AI programs to keep out. public-records-check is not one of them, and its name never changes to get around such rules.

Frankie Cleary runs it. Write to him at frankie@newonrecord.com.

How it names itself

Every request carries this user agent:

public-records-check/0.1 (+mailto:frankie@outpostagencies.com)

It never uses a web browser's user agent, and never another product's name. To write a robots.txt rule for it, use the name public-records-check.

What it reads

  • Only public pages and files: the page or file where your records are posted.
  • What you publish for programs comes first: an API, a download file, or a feed your robots.txt allows. A page made for people comes last.
  • It fills in a public search form and turns its pages the way a visitor would.
  • It doesn't follow links off the page, and it doesn't open each record's own page.
  • In its first nights on a source, it also reads older pages of the same records, a little each night, at the same pace. After that, it reads only what's new.
  • It keeps a private copy of each page it reads. It never requests a page again just to get back what it already read.

Before the nightly check began, the same name visited these sites in November 2026, to find and confirm the pages in the directory.

During those visits, faults in how the checker followed robots.txt let 20 requests reach paths that robots.txt disallowed. It misread three kinds of robots.txt file. It followed redirects to another host without reading that host's file. Once, it read a site's file from the wrong port. All were fixed on November 8, 2026, and nothing those requests returned was kept.

How fast

  • One request at a time to your site.
  • At least two seconds between the end of one request and the start of the next.
  • If your robots.txt sets a Crawl-delay longer than two seconds, it waits that long instead.
  • A night's check costs each source one or two requests, about the same as one person looking once a day.
  • If your server sent an ETag or Last-Modified header last time, the next request asks with If-None-Match or If-Modified-Since. An unchanged page then costs a 304 and nothing more.
  • Each night's check has a time limit. A source it doesn't reach in time waits for the next night. It never speeds up to fit sources in.
  • It waits up to 30 seconds for a response. It stops reading any response over 20 MB.
  • After a 5xx or a 429, it tries at most twice more. It waits as long as your Retry-After says, up to 60 seconds. Without one, it waits 5 seconds, then 20. It doesn't retry any other error.

How it reads robots.txt

  • It reads your robots.txt once a day. It follows the group for public-records-check, or the * group when there isn't one.
  • It reads the file as RFC 9309 does, so a blank line doesn't end a group. It also skips a byte-order mark at the start of the file. It reads User-agent: * Disallow: / written on one line as the two lines it means.
  • It never requests a path your robots.txt disallows for it.
  • When a request redirects to another host, it reads that host's robots.txt before it follows. It follows at most five redirects in a row.
  • A robots.txt that answers 404, or another 4xx, allows everything.
  • A robots.txt that answers a 5xx, or can't be reached, keeps it off your site that night. Night after night, that counts as a no (below).

When your site says no

Each of these counts as no: a 401, a 403, a CAPTCHA, a page that asks visitors to prove they're a person, a 429 that lasts past its retries, a secure connection that fails, or a connection closed on every attempt.

On a no, it stops. It doesn't try again with another user agent, a browser, another address or borrowed cookies.

It never signs in. It never uses a password or an API key. It never carries cookies from one site to another.

The source's page in the directory shows that night's no. A no may not last, so while the directory still reads the source, the checker comes back the next night, the same way and at the same pace.

When a site says no in its rules, or keeps turning the checker away, a person marks the source as not read, with the reason. The checker never decides that itself. After that, it opens the source's page to see whether the no still stands: on up to three nights soon after, then once a week. It opens the page only if your robots.txt allows it, and it takes no records from it. If your site answers again, a person decides whether to read the source again. To ask for that, write to frankie@newonrecord.com.

Asking it to slow down or stay away

The quickest way is robots.txt. The checker sees a change within a day.

To keep it off your whole site:

User-agent: public-records-check
Disallow: /

To slow it down, here to one request every 30 seconds:

User-agent: public-records-check
Crawl-delay: 30

To keep it off part of your site, add Disallow lines for those paths to the same group.

A group for public-records-check replaces your * group for this checker. Copy into it any Disallow lines from * that should still apply.

You can also write to frankie@newonrecord.com. Give your site's address and what you'd like: a slower pace, some pages left alone, or the whole site left alone.

Your public pages stay in the directory, with links, so people can still find them. The directory shows that they aren't read, and why.

If it caused a problem

Write to frankie@newonrecord.com with the time and the address it requested. Every request it sends is logged, so those two things are enough to find it.