XPath

XPath, short for XML Path Language, is a query language designed to navigate and select nodes in XML and HTML documents.

XPath, short for XML Path Language, is a query language designed to navigate and select nodes in XML and HTML documents. By treating a document as a hierarchical tree, XPath lets developers pinpoint specific elements, attributes, or text fragments using concise path expressions. It is a foundational technology in web scraping, browser automation, XML data processing, and countless other tasks that involve structured markup.

Whether you are extracting product prices from an e-commerce page, validating the structure of an XML feed, or writing automated UI tests with Selenium, XPath provides a reliable way to locate the exact data you need. This guide walks through the core concepts, syntax, and practical applications of XPath, with a focus on real-world usage.

The XPath Data Model

Before writing expressions, it helps to understand how XPath views a document. Rather than seeing HTML or XML as plain text, XPath interprets it as a tree of nodes. Every part of the document—elements, attributes, text, and the document itself—is represented as a node in this tree.

The main node types include:

  • Element nodes: Correspond to tags such as <div>, <p>, <book>, or <a>.
  • Attribute nodes: Represent name-value pairs inside opening tags, such as class="main" or id="content".
  • Text nodes: Contain the character data found between tags.
  • The root node: The entry point of the entire document, usually denoted by a single forward slash (/). It is distinct from the document’s top-level element.

Understanding this tree model is essential because XPath expressions are essentially paths through that tree, much like file paths in an operating system.

Core Path Expressions

XPath uses path expressions to move from one node to another. The most common operators are:

  • / — Selects from the root node or separates levels in a hierarchy.
  • // — Selects nodes anywhere in the document from the current context, regardless of depth.
  • . — Refers to the current node.
  • .. — Refers to the parent of the current node.
  • @ — Prefixes an attribute name to select an attribute node.

For example, /bookstore/book selects every book element that is a direct child of the root-level bookstore element. In contrast, //title retrieves every title element in the document, no matter how deeply it is nested.

These operators can be combined. The expression //div//a finds all anchor elements nested inside any div, skipping intermediate layers if necessary.

Filtering with Predicates

Path expressions often return more nodes than you need. Predicates, enclosed in square brackets ([...]), let you filter the result set based on position, attributes, or other conditions. XPath uses 1-based indexing, so the first item is at position 1.

Common predicate examples include:

  • //book[1] — Selects the first book element within its parent.
  • //book[last()] — Selects the last book element within its parent.
  • //book[@price] — Selects all book elements that have a price attribute.
  • //book[@price="29.99"] — Selects book elements whose price attribute equals 29.99.
  • //li[@class="item"][2] — Selects the second li element with a class of item.

Predicates can also be chained to apply multiple filters in sequence, which is useful when working with complex or irregular document structures.

Wildcards and Node Tests

Sometimes the exact element or attribute name is unknown. XPath supports wildcards for these situations:

  • * — Matches any element node. For instance, /bookstore/* returns every child element of bookstore.
  • @* — Matches any attribute node. The expression //book[@*] finds all book elements that carry at least one attribute.
  • node() — Matches any node type, including elements, text, and comments.

Wildcards are especially helpful when scraping pages generated by content management systems, where class names or tag structures may vary between templates.

Useful Built-in Functions

XPath includes a library of functions that simplify text matching, string manipulation, and logical comparisons. Some of the most frequently used functions are:

  • contains(string, substring) — Returns true if the first string contains the second. Example: //div[contains(@class, "header")] selects div elements whose class attribute includes the word "header".
  • text() — Returns the text content of a node. Example: //a[text()="Sign in"] finds links with exactly that visible label.
  • starts-with(string, substring) — Checks whether a string begins with a given prefix.
  • normalize-space(string) — Strips leading and trailing whitespace and collapses internal whitespace to single spaces, which is useful for cleaning up scraped text.

These functions can be combined inside predicates to build highly specific queries without resorting to post-processing in another language.

Practical Example: Parsing HTML with Python and lxml

In production environments, XPath is most commonly used alongside parsing libraries. Python’s lxml package offers a fast, standards-compliant implementation that integrates well with requests for web scraping workflows.

Consider the following HTML fragment:

<html>
  <body>
    <div id="container">
      <ul>
        <li class="item" data-id="1"><a href="https://example.com/1">Python Basics</a></li>
        <li class="item" data-id="2"><a href="https://example.com/2">XPath Fundamentals</a></li>
        <li class="item" data-id="3"><a href="https://example.com/3">Web Scraping with Python</a></li>
      </ul>
    </div>
  </body>
</html>

The following script demonstrates how to load this markup and run several XPath queries:

from lxml import etree

html_content = """
<html>
  <body>
    <div id="container">
      <ul>
        <li class="item" data-id="1"><a href="https://example.com/1">Python Basics</a></li>
        <li class="item" data-id="2"><a href="https://example.com/2">XPath Fundamentals</a></li>
        <li class="item" data-id="3"><a href="https://example.com/3">Web Scraping with Python</a></li>
      </ul>
    </div>
  </body>
</html>
"""

# Parse the HTML string into an element tree
tree = etree.HTML(html_content)

# Extract the text of every link
titles = tree.xpath('//ul/li/a/text()')
print("Link titles:", titles)

# Extract the href attribute of every link
links = tree.xpath('//ul/li/a/@href')
print("Link URLs:", links)

# Locate the link inside the list item whose data-id is 2
target_title = tree.xpath('//li[@data-id="2"]/a/text()')
print("Target title:", target_title)

Running this code produces structured lists of titles and URLs, and it isolates a specific item using an attribute predicate. This pattern scales directly to larger documents: load the page, craft an XPath expression, and extract the matching nodes or attribute values.

Conclusion

XPath remains one of the most direct and expressive ways to query structured documents. Its path-based syntax mirrors the logical layout of XML and HTML, while predicates, wildcards, and built-in functions provide the flexibility needed for real-world data extraction. By mastering the fundamentals covered here—node types, path operators, filtering, and common functions—you will be well equipped to handle everything from simple attribute lookups to complex scraping and automation tasks.