Skip to content
Output Parsing
BeautifulSoup

BeautifulSoup

Focused BeautifulSoup recipes: reading attributes, element text, nested finds, find_all, CSS selectors, and stripped_strings.

BeautifulSoup — read a tag attribute (CSRF token, id, href)

find(tag, {attribute: value}) searches the parsed document in order and returns the first matching element object. Indexing that object with an attribute name, such as ["value"] or ["href"], returns the corresponding HTML attribute string.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

csrf_element = soup.find("input", {"id": "csrf"})
csrf = csrf_element["value"]

name_element = soup.find("input", {"name": "csrf"})
name = name_element["value"]

link_element = soup.find("a", {"target": "_blank"})
href = link_element["href"]

button_element = soup.find("button", {"class": "delete-btn"})
file_id = button_element["value"]
file_id = file_id.strip()

Markup these calls target

<input id="csrf" name="csrf" value="9f8a1c">
<a target="_blank" href="/files/report.pdf">Open</a>
<button class="delete-btn" value="42">Delete</button>

What each variable holds

csrf -> "9f8a1c"
name -> "9f8a1c"
href -> "/files/report.pdf"
file_id -> "42"

Find by: beautifulsoup, bs4, attribute, value, csrf token, hidden input, href, find by id, find by name, find by class, scrape token, grab id · Source: PG/Monster, WSA, PG/Zipper, PG/WallpaperHub

BeautifulSoup — read element text (command output, reflected value)

find() returns the first matching element. Calling get_text() on that element joins the text nodes beneath it into a Python string, and strip() removes surrounding whitespace. get_text() describes parsed text nodes, not what a browser considers visually rendered text; CSS visibility and layout are not evaluated.

When text also contains a label such as Balance: $250, each transformation is stored separately before conversion to an integer.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

output_element = soup.find("span")
output = output_element.get_text()

result_element = soup.find("div", {"class": "divmin"})
result = result_element.get_text()
result = result.strip()

heading_element = soup.find("h4")
heading = heading_element.get_text()
heading = heading.strip()

# value embedded in a label -> split off the label, cast to a number
balance_element = soup.find("strong")
balance_text = balance_element.get_text()
balance_text = balance_text.split(": ")[1]
balance_text = balance_text.strip("$")
balance = int(balance_text)

Markup these calls target

<span>uid=33(www-data) gid=33(www-data)</span>
<div class="divmin">  root:x:0:0:root:/root:/bin/bash  </div>
<h4>config.php</h4>
<strong>Balance: $250</strong>

What each variable holds

output  -> "uid=33(www-data) gid=33(www-data)"
result  -> "root:x:0:0:root:/root:/bin/bash"
heading -> "config.php"
balance -> 250            (an int, ready for arithmetic)

Find by: beautifulsoup, bs4, get_text, element text, inner text, command output, reflected value, read response, strip, parse number, split value · Source: PG/XposedAPI, CWEE/Prototype Pollution, CWEE/Second Order, CWEE/Gift Card

BeautifulSoup — read text outside the HTML element

With the html.parser parser used below, text before or after the <html> element remains a child of the complete BeautifulSoup document rather than a child of the <html> element. Calling get_text() on soup traverses every text node in the parsed document; calling it on html_element traverses only descendants of <html>.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")
all_text = soup.get_text()

html_element = soup.find("html")
html_text = html_element.get_text()

Response body these calls parse

command output
<html><body><h1>page content</h1></body></html>

What each variable holds

all_text  -> "command output\npage content"
html_text -> "page content"

Command output written before the application template is therefore present in all_text, even though it is outside the <html> element.

Find by: beautifulsoup, bs4, get_text, text outside html, root level text, root text node, navigablestring, command output before html, in band command output

BeautifulSoup — drill into a nested element

The first find() stores a container element. Calling find() on that stored element searches only through its descendants. The class_= keyword is BeautifulSoup’s Python-safe name for the HTML class attribute because class is a reserved Python keyword.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

# store the container, then search inside it
container = soup.find("div", class_="card-content")
paragraph = container.find("p")
content = paragraph.get_text()
content = content.strip()

Markup this targets

<div class="card-content">
  <h5>report.txt</h5>
  <p>&lt;NESTED_VALUE&gt;</p>
</div>

Result

content -> "<NESTED_VALUE>"

Find by: beautifulsoup, bs4, nested find, chained find, class_, find within, parent child, drill into, container, scoped search · Source: CWEE/Second Order LFI

BeautifulSoup — find_all then pick by index

find_all(tag) returns a list of every matching element in document order. Python index 1 selects the second element because list indexes start at zero, while index -1 selects the last element.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

center_elements = soup.find_all("center")
second_element = center_elements[1]
second = second_element.get_text()

cell_elements = soup.find_all("td")
last_element = cell_elements[-1]
last = last_element.get_text(strip=True)

Markup this targets

<center>Header</center>
<center>[email protected]</center>
<table><tr><td>id</td><td>0042</td></tr></table>

What each variable holds

second -> "[email protected]"
last   -> "0042"

Find by: beautifulsoup, bs4, find_all, index, nth match, second element, last element, list of elements, td, center, in band dump · Source: CWEE/XPath in-band

BeautifulSoup — select a cell from one of multiple tables

Pages containing multiple tables can be traversed in stages: collect the tables, select one table, collect its matching rows, select one row, then select the required cell.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

tables = soup.find_all("table")
target_table = tables[1]
rows = target_table.find_all("tr", {"class": "table-active"})
target_row = rows[4]
cells = target_row.find_all("td")
value = cells[3].get_text(strip=True)

Markup this targets

<table>
  <tr><td>Unrelated table</td></tr>
</table>

<table>
  <tr class="table-active"><td>1</td><td>...</td><td>...</td><td>First row</td></tr>
  <tr class="table-active"><td>2</td><td>...</td><td>...</td><td>Second row</td></tr>
  <tr class="table-active"><td>3</td><td>...</td><td>...</td><td>Third row</td></tr>
  <tr class="table-active"><td>4</td><td>...</td><td>...</td><td>Fourth row</td></tr>
  <tr class="table-active"><td>5</td><td>...</td><td>...</td><td>&lt;TARGET_VALUE&gt;</td></tr>
</table>

Result

value -> "<TARGET_VALUE>"

Find by: beautifulsoup, bs4, find_all, multiple tables, indexed table, td, tr, indexed row, indexed cell, table traversal, scoped row search · Source: HTB/HorrorFeeds

BeautifulSoup — CSS selectors with select() / select_one()

A CSS selector describes elements by tag, class, identifier, and relationship. select() accepts a selector and returns a list of every match; select_one() returns only the first. In section.container-list-tiles > div, section is the tag, .container-list-tiles is the class selector, and > restricts the div matches to direct children of that section.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

tiles = soup.select("section.container-list-tiles > div")  # direct children -> list
first = soup.select_one("section.container-list-tiles > div")
names = []
for d in tiles:
    names.append(d.get_text(strip=True))

Markup this targets

<section class="container-list-tiles">
  <div>Laptop</div>
  <div>Mouse</div>
</section>

What each variable holds

tiles -> [<div>Laptop</div>, <div>Mouse</div>]   (2 elements)
first -> <div>Laptop</div>
names -> ["Laptop", "Mouse"]

Find by: beautifulsoup, bs4, css selector, select, select_one, direct child, combinator, class selector, query, list of nodes, scrape grid · Source: WSA SQLi in-band

BeautifulSoup — collect a whole column (find_all + loop)

find_all() collects a repeated element in document order. Reading each element’s text produces a list that can represent a table column, result set, or group of labels. strip=True removes surrounding whitespace from each text fragment before joining it.

For a content-based oracle, the same extraction runs against a controlled false request and a candidate request. A repeatable difference between the two lists becomes the Boolean signal.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

titles = []
for t in soup.find_all("h3"):
    titles.append(t.get_text(strip=True))

Markup this targets

<h3>Laptop</h3>
<h3>Mouse</h3>
<h3>Keyboard</h3>

Result

titles -> ["Laptop", "Mouse", "Keyboard"]

Find by: beautifulsoup, bs4, find_all, collect column, scrape all, get_text strip, all matches, loop, build list, oracle list, diff results · Source: WSA SQLi in-band

BeautifulSoup — separate text fragments with stripped_strings

When one element contains text beneath several descendants, get_text() joins those text nodes into one string. stripped_strings is a generator that yields one whitespace-trimmed string for each non-empty text node. Iterating over the generator retains the boundaries between a name, price, label, or other separate fragments.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

tile  = soup.find("div", {"class": "tile"})
parts = []
price = None
for t in tile.stripped_strings:
    parts.append(t)
    if price is None and t.startswith("$"):
        price = t

Markup this targets

<div class="tile">
  <h3>Laptop</h3>
  <span>$1,299</span>
  <small>in stock</small>
</div>

What each variable holds

parts -> ["Laptop", "$1,299", "in stock"]
price -> "$1,299"

Find by: beautifulsoup, bs4, stripped_strings, text nodes, fragments, multiple texts, generator, filter text, price, name and price, split element text · Source: WSA SQLi in-band

Structured Data