Lists and tables

BeautifulSoup — find_all then pick by index

find_all(tag) returns a list of every matching element in document order. Python index 1 selects the second element because list indexes start at zero, while index -1 selects the last element.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

center_elements = soup.find_all("center")
second_element = center_elements[1]
second = second_element.get_text()

cell_elements = soup.find_all("td")
last_element = cell_elements[-1]
last = last_element.get_text(strip=True)

Markup this targets

<center>Header</center>
<center>[email protected]</center>
<table><tr><td>id</td><td>0042</td></tr></table>

What each variable holds

second -> "[email protected]"
last   -> "0042"

_{Find by: beautifulsoup, bs4, find_all, index, nth match, second element, last element, list of elements, td, center, in band dump · Source: CWEE/XPath in-band}

BeautifulSoup — select a cell from one of multiple tables

Pages containing multiple tables can be traversed in stages: collect the tables, select one table, collect its matching rows, select one row, then select the required cell.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

tables = soup.find_all("table")
target_table = tables[1]
rows = target_table.find_all("tr", {"class": "table-active"})
target_row = rows[4]
cells = target_row.find_all("td")
value = cells[3].get_text(strip=True)

Markup this targets

<table>
  <tr><td>Unrelated table</td></tr>
</table>

<table>
  <tr class="table-active"><td>1</td><td>...</td><td>...</td><td>First row</td></tr>
  <tr class="table-active"><td>2</td><td>...</td><td>...</td><td>Second row</td></tr>
  <tr class="table-active"><td>3</td><td>...</td><td>...</td><td>Third row</td></tr>
  <tr class="table-active"><td>4</td><td>...</td><td>...</td><td>Fourth row</td></tr>
  <tr class="table-active"><td>5</td><td>...</td><td>...</td><td>&lt;TARGET_VALUE&gt;</td></tr>
</table>

Result

value -> "<TARGET_VALUE>"

_{Find by: beautifulsoup, bs4, find_all, multiple tables, indexed table, td, tr, indexed row, indexed cell, table traversal, scoped row search · Source: HTB/HorrorFeeds}

BeautifulSoup — collect a whole column (find_all + loop)

find_all() collects a repeated element in document order. Reading each element’s text produces a list that can represent a table column, result set, or group of labels. strip=True removes surrounding whitespace from each text fragment before joining it.

For a content-based oracle, the same extraction runs against a controlled false request and a candidate request. A repeatable difference between the two lists becomes the Boolean signal.

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

titles = []
for t in soup.find_all("h3"):
    titles.append(t.get_text(strip=True))

Markup this targets

<h3>Laptop</h3>
<h3>Mouse</h3>
<h3>Keyboard</h3>

Result

titles -> ["Laptop", "Mouse", "Keyboard"]

_{Find by: beautifulsoup, bs4, find_all, collect column, scrape all, get_text strip, all matches, loop, build list, oracle list, diff results · Source: WSA SQLi in-band}