Structured Data

Pulling structure out of responses without BeautifulSoup: JSON traversal, PDF text extraction, and zipping scraped lists.

Zip two scraped lists into a dict

When a page renders two related columns, each column can be collected into its own list. zip(names, prices) pairs the first name with the first price, the second name with the second price, and so on. dict() converts those pairs into a dictionary in which each name is a key and its corresponding price is the value.

The pairing is positional rather than based on HTML relationships. Both lists must therefore be collected in the same document order and contain the same number of relevant entries.

names = []
for t in soup.find_all("h3"):
    names.append(t.get_text(strip=True))

prices = []
for d in soup.select("section.list > div"):
    for t in d.stripped_strings:
        if t.startswith("$"):
            prices.append(t)

item_pairs = zip(names, prices)
items = dict(item_pairs)
for name, price in items.items():
    print(f"{name}: {price}")

Sample markup

<section class="list">
  <div><h3>Laptop</h3><span>$1,299</span></div>
  <div><h3>Mouse</h3><span>$25</span></div>
</section>

Example output

Laptop: $1,299
Mouse: $25

_{Find by: zip, dict, combine lists, pair, names and prices, mapping, dictionary · Source: WSA SQLi in-band}

Extract text from a downloaded PDF (pypdf)

A generated PDF is a binary response rather than HTML text. Path.write_bytes() first stores those bytes as a local artifact. PdfReader parses the PDF structure, reader.pages exposes each page object, and page.extract_text() returns the text content pypdf can recover from that page.

Extracted text follows the PDF’s internal text objects rather than necessarily matching its visual layout. Tabs are normalized to spaces before parsing. A regular expression with re.DOTALL then isolates text between two stable labels; DOTALL allows the . wildcard to cross page-extracted newline characters.

from pypdf import PdfReader
from pathlib import Path
import re

# download, then persist the binary response
r = s.get(url=f"{URL}/<PDF_ENDPOINT>", verify=False, timeout=10, proxies=PROXIES)
pdf_file = Path("output.pdf")
pdf_file.write_bytes(r.content)

reader = PdfReader(pdf_file)
pdf_text = ""
for page in reader.pages:
    page_text = page.extract_text()
    pdf_text += page_text + "\n"
pdf_text = pdf_text.replace("\t", " ")

# isolate the text between two stable labels
match = re.search(r"<START_LABEL>\s*(.*?)\s*<END_LABEL>", pdf_text, re.DOTALL)
if match:
    output = match.group(1)
    output = output.strip()
    print(output)
else:
    print("[-] output not found in PDF")

_{Find by: pdf, pypdf, PdfReader, extract_text, download pdf, certificate, invoice, ssti pdf, cmdi pdf, command output, parse pdf, pages, pure python, write_bytes · Source: CWEE/PDF RCE}

Parse JSON responses — r.json() + staged .get traversal

r.json() parses the response body and returns the corresponding Python structure: a JSON object becomes a dictionary, a JSON array becomes a list, and primitive JSON values become their Python equivalents. Invalid JSON raises requests.exceptions.JSONDecodeError.

dictionary.get("key") returns the stored object or None when the key is absent. A second argument supplies another default. Storing each nested level in a separate variable makes its type visible before continuing. A list must contain an element before index 0 can be read. The explicit loop stores the first dictionary whose method key contains GET.

json.loads(text) performs the same parsing for an existing string. json.dumps(obj, indent=2) performs the opposite operation and formats a Python object as readable JSON text.

import json

r = s.get(url=POLL_URL, verify=False, proxies=PROXIES, timeout=10)
data = r.json()                              # JSON body -> dict/list (raises on non-JSON body)

token  = data.get('uuid')                    # safe key read: None if absent (no KeyError)
status = data.get('status', 'unknown')       # ...with a fallback default
meta   = data.get('meta', {})
host   = meta.get('host')

# value buried in a list-of-objects -- guard the empty list BEFORE indexing [0]:
reqs  = data.get('data', [])
if reqs:
    first = reqs[0]
else:
    first = {}
query = first.get('query', {})
out   = query.get('data')   # e.g. the exfiltrated value

# first list item matching a predicate (None if none match):
hit = None
for x in reqs:
    if x.get('method') == 'GET':
        hit = x
        break

# parse a JSON *string*, and pretty-print any object to eyeball its shape:
obj = json.loads(r.text)
print(json.dumps(obj, indent=2))

Sample JSON body the calls target

{
  "uuid": "1b2c3d4e",
  "status": "ok",
  "data": [
    { "method": "GET", "query": { "data": "726f6f74" } }
  ]
}

What each variable holds

token  -> '1b2c3d4e'
status -> 'ok'
host   -> None                  (no 'meta' key, default {} then .get -> None)
out    -> '726f6f74'            (data[0]['query']['data'])
hit    -> {'method': 'GET', 'query': {...}}

_{Find by: json, parse json, r.json, response json, api response, dict, get key, default value, keyerror, indexerror, jsondecodeerror, nested, traversal, list of objects, index guard, first match, find in list, predicate, json.loads, json.dumps, pretty print, navigate, extract field, webhook, poll, oob exfil · Source: HTB/VoidWhispers}

BeautifulSoup String partition