Function reference

This page documents the functions shipped in src/gdeltnews/ (and the most important helpers they rely on). It focuses on behavior, inputs, outputs, and notable edge cases.

Top-level package exports (gdeltnews)

download(start, end, *, outdir="gdeltdata", overwrite=False, decompress=True, timeout=30, show_progress=True, workers=8, retries=3) -> DownloadStats

Download GDELT Web NGrams minute files for an inclusive time range.

  • Inputs
    • start, end: either datetime or a timestamp string. Supported string formats include:
      • YYYY-MM-DDTHH:MM:SS (optionally with trailing Z)
      • YYYY-MM-DD HH:MM:SS
      • YYYYMMDDHHMMSS
    • outdir: destination directory for downloaded files
    • overwrite: redownload even if the .gz already exists locally
    • decompress: if True, also write the decompressed .json files. Reconstruction reads the .gz directly, so leaving this at False halves both disk usage and I/O.
    • timeout: HTTP timeout in seconds
    • show_progress: show a progress bar across minute slots
    • workers: how many minute files to fetch concurrently. This step is I/O bound, so threads are used. Pass 1 for the old strictly-serial behaviour.
    • retries: extra attempts per file on network errors and 5xx responses
  • Outputs
    • DownloadStats(requested_minutes, downloaded_gz, decompressed_json, missing, failed). missing and failed are deliberately distinct: missing means GDELT publishes no file for that minute (normal), while failed means the request kept erroring after every retry and there is a genuine gap in the downloaded range.
  • Behavior
    • Covers every minute from start to end (inclusive), fetching <YYYYMMDDHHMMSS>.webngrams.json.gz from the GDELT Web NGrams base URL.
    • All requests share one requests.Session, so the connection is reused instead of re-handshaking per file.
    • A 404 (or any other 4xx) is treated as “no file for this minute” and is not retried. Network errors and 5xx are retried with exponential backoff, then counted in failed.
    • Each file is written to <name>.gz.part and renamed on success, so an interrupted run can never leave a truncated .gz that a later run mistakes for complete.

fetch_minute(ts, dest_dir, *, overwrite=False, timeout=30, quiet=False, session=None, retries=3) -> (path | None, outcome)

Lower-level variant of download_gdelt_file that also returns why a minute produced no file. outcome is "ok", "cached", "missing" (GDELT publishes nothing for that minute) or "failed" (still erroring after every retry).

Use download() unless you are driving your own loop and need per-minute outcomes — the GUI uses this to advance its progress bar while still reporting how many slots genuinely failed.

reconstruct(input_dir="gdeltdata", output_dir="gdeltpreprocessed", *, language=None, url_filters=None, processes=None, delete_gz=False, delete_empty_csv=True, show_progress=True, keep_unmerged=False) -> None

Bulk reconstruction runner for a folder of GDELT *.webngrams.json.gz files.

  • Inputs
    • input_dir: directory containing the .gz files
    • output_dir: directory where per-input CSVs are written
    • language: optional language code filter (e.g. "it"). None keeps all languages
    • url_filters: optional iterable of URL substrings to keep (any match keeps a URL)
    • processes: number of worker processes used per input file (None uses all cores)
    • delete_gz: delete original .gz after processing
    • delete_empty_csv: delete CSVs that contain only the header row
    • show_progress: show a progress bar across files
    • keep_unmerged: if True, fragments that never overlapped anything are appended to the article instead of being dropped. Default False, which reproduces the historical output exactly.
  • Outputs
    • Writes one CSV per input file to output_dir with delimiter | and header:
      • Text|Date|URL|Source
  • Behavior
    • Streams each .json.gz input directly; it does not create, require, or delete decompressed .json files.

filtermerge(input_dir, output_file, *, query=None, keep_temp=False, verbose=True) -> None

Filter, merge, and deduplicate reconstructed CSVs.

  • Inputs
    • input_dir: directory of CSV files (as produced by reconstruct)
    • output_file: destination CSV path
    • query: Boolean query string using AND, OR, NOT plus parentheses. Use double quotes for phrases, e.g. "julia roberts". A term containing anything other than letters/digits/underscore must be quoted — write "centro-destra", not centro-destra. Every term needs an explicit operator between it and the next.
    • keep_temp: keep the intermediate output_file + ".tmp" file
    • verbose: print progress messages
  • Behavior
    • Performs case-insensitive whole-word matching over the Text column: a query for butti matches Butti but not debutti, buttiamo or Gabutti. Word boundaries are only required on sides that begin/end with a word character, so "c++" and ".net" still match.
    • Deduplicates by URL, keeping the row with the longest Text.

Module: gdeltnews.wordmatch

transform_dict(original_dict: dict[str, list[dict]]) -> dict[str, list[Entry]]

Convert raw per-URL entry dictionaries into simplified “sentence fragment” entries.

  • Builds sentence from pre, ngram, post.
  • Normalizes some early-position artifacts (e.g. keeps substring after " / " when pos < 20).

reconstruct_sentence(fragments: list[str], positions: list[int] | None = None, keep_unmerged: bool = False) -> str

Reconstruct a longer text by merging overlapping fragments (word overlap).

  • Greedy overlap merge (max overlap first).
  • If positions are given, enforces a constraint that prevents obviously wrong reorderings:
    • only appends fragments whose position is not earlier than the current max position
    • only prepends fragments whose position is not later than the current min position
  • Each round selects the fragment with the largest overlap; ties go to the lowest index, and within one fragment an append beats a prepend of equal length.
  • When no remaining fragment overlaps, the merge stops and the leftovers are dropped — an article can therefore come out truncated. keep_unmerged=True appends them in position order instead.

Internally the candidate overlaps are indexed by their first/last word rather than tried one length at a time, and fragments that can no longer be placed are pruned. This is a pure speedup (measured ~2-3x on real files): the output is identical to the original quadratic scan, which tests/test_reconstruct_equivalence.py pins down by fuzzing both against each other.

remove_overlap(text: str) -> str

Conservative cleanup that removes simple duplicated prefix/suffix overlaps in the final reconstructed text.

load_and_filter_data(input_file: str, language_filter="en"|None, url_filter=None) -> (articles: dict, url_order: list[str])

Read a *.webngrams.json or *.webngrams.json.gz file line-by-line and:

  • optionally filter by language (language_filter=None keeps all)
  • optionally filter by URL substring(s) — empty/whitespace-only filters are dropped, so passing [""] no longer matches every URL
  • skip the raw-byte pre-filter when any URL filter contains non-ASCII characters: GDELT may emit the URL with \uXXXX escapes, so the filter’s UTF-8 bytes need not appear in the raw line even when the decoded URL matches. Such runs are slightly slower but never lose data
  • transform each kept record into a sentence-level entry inline (one pass over the file)
  • group entries by URL
  • return:
    • the per-URL entries dict
    • the URL order as first encountered in the file (kept for backward compatibility; process_file_multiprocessing no longer relies on it)

determine_source_label(url: str, url_filters: list[str] | None = None) -> str

Derive a Source label from URL filters:

  • exactly one filter matches: returns that filter string
  • multiple filters match: returns "Multiple URL matched"
  • no filters or no matches: returns ""

process_article((url, entries), url_filters=None, keep_unmerged=False) -> dict[str, str]

Reconstruct one article (intended for multiprocessing).

  • sorts entries by pos (without mutating the caller’s list)
  • merges fragments with reconstruct_sentence
  • runs remove_overlap and basic output cleanup
  • returns a dict with url, text, date, source

url_filters accepts either a list[str] (back-compat — casefolded per call) or a list of pre-casefolded (original, casefolded) tuples. process_file_multiprocessing passes the tuple form so the casefold work happens once per file rather than once per article.

If reconstruction raises, the exception is caught inside process_article so a single bad article cannot kill the multiprocessing worker. The returned dict gains an extra "error" key with a short "ExceptionType: message" string; url and source are still populated so the driver can log which article failed.

process_file_multiprocessing(input_file, output_file, language_filter="en"|None, url_filter=None, num_processes=None, pool=None, show_progress=True, keep_unmerged=False) -> None

Core driver for reconstructing one JSON or JSON.GZ file into a CSV.

  • Uses a multiprocessing pool with imap_unordered.
  • Streams results to disk as workers finish — the output rows therefore appear in worker-completion order, not in original URL order. This keeps memory flat regardless of input size (no in-memory buffer of all reconstructed articles).
  • Always writes a CSV header even when no articles are found.
  • Per-article reconstruction failures are logged to stdout (or via tqdm.write when the progress bar is active) and the failing article is written with empty Text but a populated URL/Source, so it is still traceable in the output CSV.
  • New optional kwargs:
    • pool: an existing multiprocessing.Pool to reuse. When this driver is called once per input file (as it is from reconstruct), passing one pool across all calls avoids the per-file pool-startup cost (significant on Windows where Pool startup re-imports the module).
    • show_progress: set to False to silence the per-file tqdm bar (useful when an outer caller has its own progress bar).
    • keep_unmerged: forwarded to reconstruct_sentence; see above.

Input files are opened with isal.igzip when the optional fast extra is installed (pip install "gdeltnews[fast]") and with the standard library gzip otherwise. The fallback is automatic and changes nothing but speed.