Dosieto
A record of how Bulgarian property listings change
A crawler saves Sofia property listings every day, so there is a record of how prices and details change, and of listings that are deleted and posted again to look new. The listing site only shows the date of the last edit; Dosieto keeps every version.
dosieto.demodomain.site
- Role
- Solo: crawler, data pipeline, database, web
- Platforms
- Data pipeline · Web: read-only preview
- Tech
- TypeScriptNode.jsPostgreSQLPlain SQLgot-scrapingcheerioSimHashsharpZodNext.js 16Docker
10k+
Sofia listings archived and growing
2 s
minimum between requests, one at a time
17
forward-only SQL migrations
0
phone numbers, names or descriptions stored
The problem
A listing only shows its last edit
Property sites in Bulgaria show when a listing was last “corrected”, but not what changed. Prices drop and rise silently, and a listing can be deleted and posted again so that it looks new on the market.
Dosieto reads each listing, records its price and parameters, and reads it again after every correction. The result is a timeline per listing and statistics for the whole market, written strictly as observations, never as accusations.
01 · Archive
Every version of every listing
Each listing gets a page with its timeline: when it was first seen, every price change, every new “corrected” date and whether anything actually changed, and when it disappeared from the site.
- How long a listing has really been on the market, from its number
- Corrections with no real change are counted separately
- Price per m² compared with the neighbourhood median
02 · Live
A feed of what changed in the last hour
The sitemap is crawled every hour and pages are read in order, politely. The live page shows each observation as it is recorded: first reads, price changes, real and empty corrections, new photos and removals.
- Filters by type of change
- Crawler status, coverage and the next scheduled run
03 · Market
Statistics no listing site shows
Because every reading is stored, the archive can answer questions about the whole market: median price per m² by neighbourhood, median prices and rents by property type, and how many listings have been on the market for over a year.
04 · Matching
Spotting the same property posted again
Relisted properties are matched across different listing numbers using a SimHash signature of the description, perceptual hashes of the photos and weighted attributes such as area, floor and neighbourhood. Precision is measured per tier against hand labels before any match is shown.
- Shows what matches and what differs, with a confidence score
- In progress: matching is the current phase of the project
Under the hood
How it's built
- 01
A polite, resumable crawler
got-scraping and cheerio read the sitemap and listing pages with a 2-second gap and a single connection, stop on the first 403 or 429, and resume from a checkpoint.
- 02
Privacy rules in code
Contact blocks are stripped before raw HTML is archived, and the run fails loudly if a phone number or email survives. Descriptions are kept only as a SimHash signature; images are capped at 400px.
- 03
Plain SQL on PostgreSQL
No ORM: typed queries with the postgres driver, numbered forward-only migrations, Zod at every parse boundary, timestamps stored in UTC and shown in Sofia time.
- 04
Scheduled on the Mac, previewed on a server
launchd runs the crawls and backups locally; a read-only snapshot of the database is pushed to a Docker Compose preview behind Caddy, served by a SELECT-only database user.
- 05
Built with an agent, under rules
Developed with Claude Code from a written spec with hard rules for network and data discipline, so the agent can't speed up the crawler or store personal data by accident.