Skip to content

Auto-translated map_item updates from 4CAT: weibo - #108

Draft
4cat-to-zeeschuimer-automation-pr[bot] wants to merge 1 commit into
masterfrom
auto/4cat-map-item-sync-weibo
Draft

4cat-to-zeeschuimer-automation-pr[bot] wants to merge 1 commit into
masterfrom
auto/4cat-map-item-sync-weibo

Conversation

@4cat-to-zeeschuimer-automation-pr

Copy link
Copy Markdown
Contributor

🤖 This PR was auto-generated by the 4CAT map_item sync workflow. The JavaScript was produced by an LLM and requires human review before merging — including manual fixes for any lint warnings flagged below.

Generation parameters

  • Model: gpt-oss-120b (provider: litellm)
  • LLM server: https://llmproxy.uva.nl
  • Total LLM time: 25.14s
  • Trigger: push d800a91...9afb2dc to 4CAT master (module: weibo).

Summary

  • ✅ 1 translated
  • ⚠️ 1 translated with lint warnings (require manual fix)
  • ❌ 0 failed
  • ❔ 0 skipped
Datasource Module Time Warnings
datasources/weibo/search_weibo.py modules/weibo.js 25.14s ⚠️ 3

⚠️ Lint warnings — fix before merging

The following datasources translated successfully but the static lint flagged issues that need human fixes. The auto-generated code was spliced into the JS module as-is; please patch the file directly in this PR.

datasources/weibo/search_weibo.py -> modules/weibo.js

  • [map_item_function] Even without an f prefix, "text {var}" / 'text {var}' are literal text in JavaScript — no interpolation happens. Whenever the original Python used an f-string, the JS must use a template literal (backticks).
  • [map_item_function] Regex detected. The current LLM translates regex unreliably (escapes, character classes, flags) — please verify the regex behavior against the Python original by hand.
  • [helpers_to_add[4]] Regex detected. The current LLM translates regex unreliably (escapes, character classes, flags) — please verify the regex behavior against the Python original by hand.

datasources/weibo/search_weibo.py -> modules/weibo.js (25.14s) ⚠️

LLM commentary:

Implemented all helper functions that the original Python map_item relied on: weibo_is_post, weibo_is_shortened, weibo_first_set, weibo_post_text, weibo_hashtags, weibo_image_urls, and weibo_video_url. Used global helpers (MappedItem, MissingMappedField, MapItemException, readable_url, strip_tags, value_or_missing, py_get, formatUtcTimestamp) as provided by Zeeschuimer. Date parsing uses Date.parse which handles the original format with timezone. All Python f‑strings become template literals, and boolean "yes"/"no" strings are reproduced. Missing fields use new MissingMappedField('') to match Python's MissingMappedField(''). No additional imports are required.

4CAT commits:

  • 9afb2dc weibo: pass referrer to downloaders (fixes image/video direct downloads)
Python diff
diff --git a/datasources/weibo/search_weibo.py b/datasources/weibo/search_weibo.py
index 6a1d38cd..077b601e 100644
--- a/datasources/weibo/search_weibo.py
+++ b/datasources/weibo/search_weibo.py
@@ -32,6 +32,10 @@ class SearchWeibo(Search):
         "[Zeeschuimer browser extension](https://github.com/digitalmethodsinitiative/zeeschuimer)"
     ]
 
+    # Weibo's image and video servers refuse requests from a browser that does
+    # not say it came from Weibo
+    media_request_headers = {"Referer": "https://weibo.com/"}
+
     def get_items(self, query):
         """
         Run custom search

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant