Writing· · 2 min read
The library has no API
Scraping is what you do when an institution has decided your automation is not their problem. Notes from building a service on top of a website that never agreed to it.
- scraping
- interface
- reliability
The Oulu library has my books, a renewal button, and no API. Renewing on time is trivial for anyone who never forgets. For everyone else it is a small recurring fine — a tax on being human, collected weekly.
So FinnaRenewer logs in on a schedule and presses the button. It works. It is also the most fragile thing I run, and the reasons are worth writing down.
Scraping is an interface of last resort
An API is a promise: these fields, this shape, deprecated with notice. A web page is not a promise. It is a rendering, produced for a human, free to change on a Tuesday because a designer preferred more whitespace.
Building on a rendering means every selector is an assumption about someone else’s markup, and every deploy on their side is a coin flip. That is not a problem to be engineered away. It is the actual cost of the decision, and the alternative was not building it.
What you can engineer is the failure mode.
Fail loudly, on a schedule you chose
Two rules did most of the work.
Loud, not silent. A scraper that silently returns zero results is worse than one that crashes, because a service that reports success while renewing nothing is a service that quietly generates fines. Parse failures are errors. Zero loans found when loans exist is an error. The only acceptable silent path is a run that had genuinely nothing to do.
Weekly, not daily. Renewal windows are wide. Running daily would mean six no-op requests a week against someone else’s server for no benefit. Politeness toward a service you depend on is an engineering property, not a courtesy — the fastest way to lose an unofficial integration is to make it expensive for the people hosting it.
Other people’s credentials change the calculus
The moment this became multi-user, the blast radius stopped being my own account. That pushed a few things that would otherwise have been optional:
- Scraping happens server-side only. The browser never holds a library session.
- Application auth is separate from library credentials, handled by Auth.js, with bcrypt on the application side.
- Storage is Postgres with real migrations rather than a convenient blob, because credential handling is exactly where you want schema history you can read.
None of that makes storing someone else’s library password good. It makes it defensible, which is a different and lower bar, and the honest framing.
The part that annoys me
This entire project exists because an institution decided that automating your own relationship with your own loans is not their problem. The data is mine. The action is mine. The only thing missing is a documented way to ask.
Every scraper is a small monument to an interface that refused to be one.