Text extraction

BeautifulSoup

Text extraction

BeautifulSoup — read element text (command output, reflected value)

find() returns the first matching element. Calling get_text() on that element joins the text nodes beneath it into a Python string, and strip() removes surrounding whitespace. get_text() describes parsed text nodes, not what a browser considers visually rendered text; CSS visibility and layout are not evaluated.

When text also contains a label such as Balance: $250, each transformation is stored separately before conversion to an integer.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

output_element = soup.find("span")
output = output_element.get_text()

result_element = soup.find("div", {"class": "divmin"})
result = result_element.get_text()
result = result.strip()

heading_element = soup.find("h4")
heading = heading_element.get_text()
heading = heading.strip()

# value embedded in a label -> split off the label, cast to a number
balance_element = soup.find("strong")
balance_text = balance_element.get_text()
balance_text = balance_text.split(": ")[1]
balance_text = balance_text.strip("$")
balance = int(balance_text)

Markup these calls target

<span>uid=33(www-data) gid=33(www-data)</span>
<div class="divmin">  root:x:0:0:root:/root:/bin/bash  </div>
<h4>config.php</h4>
<strong>Balance: $250</strong>

What each variable holds

output  -> "uid=33(www-data) gid=33(www-data)"
result  -> "root:x:0:0:root:/root:/bin/bash"
heading -> "config.php"
balance -> 250            (an int, ready for arithmetic)

_{Find by: beautifulsoup, bs4, get_text, element text, inner text, command output, reflected value, read response, strip, parse number, split value · Source: PG/XposedAPI, CWEE/Prototype Pollution, CWEE/Second Order, CWEE/Gift Card}

BeautifulSoup — read text outside the HTML element

With the html.parser parser used below, text before or after the <html> element remains a child of the complete BeautifulSoup document rather than a child of the <html> element. Calling get_text() on soup traverses every text node in the parsed document; calling it on html_element traverses only descendants of <html>.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")
all_text = soup.get_text()

html_element = soup.find("html")
html_text = html_element.get_text()

Response body these calls parse

command output
<html><body><h1>page content</h1></body></html>

What each variable holds

all_text  -> "command output\npage content"
html_text -> "page content"

Command output written before the application template is therefore present in all_text, even though it is outside the <html> element.

_{Find by: beautifulsoup, bs4, get_text, text outside html, root level text, root text node, navigablestring, command output before html, in band command output}

BeautifulSoup — separate text fragments with stripped_strings

When one element contains text beneath several descendants, get_text() joins those text nodes into one string. stripped_strings is a generator that yields one whitespace-trimmed string for each non-empty text node. Iterating over the generator retains the boundaries between a name, price, label, or other separate fragments.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

tile  = soup.find("div", {"class": "tile"})
parts = []
price = None
for t in tile.stripped_strings:
    parts.append(t)
    if price is None and t.startswith("$"):
        price = t

Markup this targets

<div class="tile">
  <h3>Laptop</h3>
  <span>$1,299</span>
  <small>in stock</small>
</div>

What each variable holds

parts -> ["Laptop", "$1,299", "in stock"]
price -> "$1,299"

_{Find by: beautifulsoup, bs4, stripped_strings, text nodes, fragments, multiple texts, generator, filter text, price, name and price, split element text · Source: WSA SQLi in-band}

Attributes Nested elements and selectors