Add source maiambiente_pt (Maia, Portugal) - #7230
Conversation
Maiambiente is the municipal waste operator of Maia, Portugal. No login is required and no OCR is involved. The public lookup resolves street -> house number -> collection circuit -> yearly PDF. The body of that PDF is a bitmap, but it does not need OCR: every day is a flat-coloured circle on a regular grid, so the date comes from the geometry (month block, weekday row, week column) and the waste stream comes from the fill colour, taken from the calendar's own legend. The numbers printed inside the circles are never read. A day can carry two streams, which is how holiday compensations are drawn; both are returned. The calendar belongs to the circuit rather than to the address and covers a whole year, so it is fetched once per year and cached. Every Maiambiente path carries the year, so the address is re-resolved per year - one year's internal id is not necessarily valid in the next - and the next year's calendar is picked up automatically once the current one nears its end. Two things worth flagging for review: * This adds Pillow to the requirements. It is used only to decode the PDF's embedded bitmap, which arrives as /FlateDecode with /Predictor 15 - exactly a PNG IDAT - so the stream is wrapped in a valid PNG and handed to Pillow; no poppler, no MuPDF, no OCR engine. Home Assistant Core already ships Pillow as a base requirement, so this costs nothing at install time. * Maiambiente's servers send an incomplete TLS chain: they omit the intermediate that links their leaf certificate to the GlobalSign root. Browsers recover via AIA fetching, Python does not, so requests fails with CERTIFICATE_VERIFY_FAILED. The service module supplies that intermediate itself. Certificate verification stays fully enabled - this is not verify=False - and the pinned certificate is documented with instructions for refreshing it. TEST_CASES use two public buildings (Maia city hall and Maia public library) rather than private addresses, since they run on every scheduled source test. test_sources.py -s maiambiente_pt: found 279 entries for Câmara Municipal da Maia found 227 entries for Biblioteca da Maia Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The embedded bitmap is 6000x4161 (25 megapixels, 71 MB decompressed) and was
being copied twice for no reason: convert("RGB") was called on an image that
already decodes to RGB, since the PNG wrapper we build around the stream is
colour type 2, and every candidate image had all 25 million pixels walked by
getcolors() just to decide which one was the calendar.
Scoring now runs on a 1/8 subsample with NEAREST, which keeps the flat fill
colours exact, and rejected candidates are released instead of staying alive
for the rest of the loop.
Measured on a real calendar PDF with Pillow 12.3, peak process RSS drops from
214 MB to 134 MB over a 19.5 MB baseline, and RSS returns to baseline after
each read. This matters because the parse runs inside the Home Assistant
process: on a 2 GB installation the old peak was enough for the kernel to kill
Home Assistant Core outright.
Behaviour is unchanged. test_sources.py still returns 279 and 227 entries for
the two test cases, and the repository's own unit tests pass on Pillow 9.4 and
12.3.
…examples Memory ------ The large bitmap in these PDFs is not the calendar. A Maiambiente PDF carries two images, and the 6000x4161 one (71 MB decompressed) contains no month bars at all; the calendar is the 2303x1547 one. The old selection decompressed the large image purely to score it by colour and then discarded it. Selection is now structural and runs smallest image first, with "did the 12 month bars turn up" as the test. Image.open is lazy, so enumerating candidates costs only their headers and the large image is normally never decoded. Colour classification no longer builds a full RGB image per palette entry either. The bands are split once and each colour is handled with 256-entry lookup tables, one at a time, releasing each mask before the next. The result is a one-byte-per-pixel label map, after which the RGB image and its bands are dropped. That also removes a colour cache that grew without bound. Peak process RSS on a real calendar PDF with Pillow 12.3 goes from 134 MB to 62 MB over a 19.8 MB baseline, and returns to baseline after each read. This matters because the parse runs inside the Home Assistant process. Examples -------- The documentation and HOW_TO_GET_ARGUMENTS_DESCRIPTION used a private residential address as their example. They now use public buildings, the same ones the test cases use, and the preposition example uses "Vieira Carvalho" against "Vieira de Carvalho", verified live as 3 results against 0. Behaviour is unchanged: test_sources.py still returns 279 and 227 entries, and pytest tests/ passes.
…n it The signature helper walks every stream in the PDF to find the embedded PNG's name and the circuit codes. It decompressed each one, the 6000x4161 bitmap included, which inflates to roughly 75 MB. The "does it contain Tj or TJ" guard always passes on that much binary noise, so the parenthesised-text regex then ran over it, matched millions of times, and kept every match as a Python object. On a real 960 KB calendar PDF that peaked at 1234 MB and took 19.8 seconds, producing 22.8 million characters of garbage. Since this runs inside the Home Assistant process, on a 2 GB installation it means the kernel OOM killer takes Home Assistant down; the reporter's kernel journal shows the victim at 1.6 GB. Decompression is now bounded to 2 MiB per stream with decompressobj, and any stream that does not fit is skipped as an image. Content streams in these calendars are a few kilobytes. After the fix that helper peaks at 26.6 MB in 0.0 s and returns the same signature. A full read, signature plus calendar, peaks at 61 MB.
|
Updated and ready for review. Since opening this I found and fixed a serious memory problem in my own code, so the description above has been rewritten to describe the final state rather than the original one. The short version: the signature helper was decompressing every stream in the PDF to look for the embedded PNG's name, the 25 megapixel bitmap included, and then running a text regex over 75 MB of binary noise. On a 960 KB PDF that peaked at 1234 MB and took 19.8 seconds. Since the parse runs inside the Home Assistant process, that was enough for the kernel OOM killer to take Home Assistant down repeatedly on a 2 GB installation. Decompression is now bounded per stream, calendar selection is structural rather than colour-scored so the large image is normally never decoded at all, and colour classification no longer builds a full RGB image per palette entry. A full read now peaks at 61 MB over a 19.8 MB baseline. I would rather have caught that before submitting. Flagging it explicitly because "runs inside the HA process" makes footprint a correctness question, not a tuning one, and a reviewer should not have to discover it.
|
|
Since we are currently maintaining both v2 and the v3 beta, I think this might be a good candidate to target v3 only. We are not generally freezing v2 yet, but this source introduces significantly more complexity than a typical source: a custom PDF/image parser, Pillow usage and the TLS certificate workaround. Pillow is already part of the v3 dependencies, and avoiding this additional complexity in both branches would reduce the maintenance burden while we move towards v3. So unless there is a strong reason to have this available in v2 as well, I would prefer targeting release/3.0.0. @markvp What do you think? |
2 participants
Service provider
Maiambiente, E.M., the municipal waste operator of Maia, Portugal (~140k inhabitants). Public lookup at https://servicos.maiambiente.pt/cal2026/, no login.
This is the second Portuguese source in the repo, after
cm_lisboa_pt.Files added
custom_components/waste_collection_schedule/waste_collection_schedule/source/maiambiente_pt.pycustom_components/waste_collection_schedule/waste_collection_schedule/service/Maiambiente.pydoc/source/maiambiente_pt.mdAlso modified:
custom_components/waste_collection_schedule/manifest.jsonandrequirements.txt, to addPillow. See below.No changes to
README.md,info.md,sources.json,source_metadata.json,translations/*.jsonordoc/ics/*.md.How it works
The lookup chains street to house number to collection circuit to a yearly PDF. The body of that PDF is a bitmap, but it needs no OCR: every day is a flat-coloured circle on a regular grid, so
The numbers printed inside the circles are never read. The grid is anchored on the dark green month-name bars, which are always present, so circuits that skip a weekday do not break the alignment. A day can carry two streams, which is how holiday compensation is drawn, and both are returned.
The calendar belongs to the circuit rather than the address and covers a whole year, so it is fetched once per year and cached. Every Maiambiente path carries the year (
cal2026/cal2026.php,type=arruamento2026, and so on), so the address is re-resolved per year, since one year's internal id is not necessarily valid in the next. Once the current calendar nears its end the next year's is picked up automatically, which covers the December to January transition.Two things I would like reviewed explicitly
1. This adds
Pillowto the requirements.It decodes the PDF's embedded bitmap and nothing else. That bitmap arrives as
/FlateDecodewith/Predictor 15, which is exactly how a PNGIDATis encoded, so the stream is wrapped in a valid PNG header and handed to Pillow. No poppler, no MuPDF, no OCR engine.Home Assistant Core already ships
Pillowas a base requirement, so this costs nothing at install time for HA users.2. Maiambiente's servers send an incomplete TLS chain.
They present only the leaf certificate and omit the intermediate (
GlobalSign GCC R6 AlphaSSL CA 2025) linking it toGlobalSign Root CA - R6. Browsers recover by fetching it from the certificate's AIA extension, Python does not, sorequestsfails withCERTIFICATE_VERIFY_FAILEDand nothing works at all.The service module supplies that intermediate itself through an
SSLContextmounted on the session, the same shape as the existingservice/SSLError.pyworkaround. Certificate verification stays fully enabled: the chain still has to terminate at the already trusted GlobalSign root and the leaf still has to be valid for the host. This is the opposite ofverify=False. The pinned certificate is documented inline with its expiry (2027-05-21) and instructions for refreshing it. I am reporting the misconfiguration to Maiambiente; until they fix it, the source cannot connect without this.Memory
Since this parse runs inside the Home Assistant process, I treated its footprint as a correctness concern rather than an optimisation, and the later commits in this PR are that work.
Three things were wrong and are fixed:
A full read, signature plus calendar, now peaks at 61 MB over a 19.8 MB baseline and returns to baseline afterwards.
Test cases
TEST_CASESdeliberately uses two public buildings, Maia city hall and Maia public library, rather than private addresses, since these run on every scheduled source test. They also exercise the non-numeric house number case, which Maiambiente uses for named buildings.Checks
python -m pytest tests/gives 39 passedruff checkandruff format --checkare clean on all changed filestest_sources.py -s maiambiente_pt:14 January shows the two-streams-on-one-day case on live data.
Beyond the test cases, this has been running on a real Home Assistant OS installation (2 GB, HA Core 2025.12.1) since today, resolving a third address and producing correct next-collection dates.
Notes
SOURCE_CODEOWNERSis@CargoTechSolutions; I will maintain it.The service module is generated from the library in CargoTechSolutions/ha-maiambiente, which also ships this as a standalone HACS integration. That repo has a CI job asserting the generated file stays in sync, and unit tests that read a synthetic calendar drawn with the same geometry, including two-stream days, suspended days and a circuit collecting every day, so the parser is covered without publishing a real address's PDF.