Build a Java 25 Web Scraper with HtmlUnit 5

HtmlUnit is a headless browser implemented in Java. It can load HTML, maintain cookies, follow links, submit forms, execute JavaScript and expose the resulting document through a DOM API.

That makes HtmlUnit useful for browser-oriented testing, automation and web scraping where a plain HTTP client and HTML parser are not enough.

This tutorial modernizes the original example for Java 25 and HtmlUnit 5.3.0. The scraper will:

• load a web page;
• print the page title;
• extract an author with a CSS selector;
• iterate through every anchor element; and
• print output with Java 25's IO.println().

HtmlUnit 5 Maven dependency

Older HtmlUnit applications used the Maven group ID net.sourceforge.htmlunit and Java packages beginning with com.gargoylesoftware.htmlunit. Modern HtmlUnit uses org.htmlunit for both the Maven coordinates and Java packages.

The current HtmlUnit dependency is:

<dependency>
    <groupId>org.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>5.3.0</version>
</dependency>

Java 25 Maven POM

For Java 25, use Maven's release setting instead of the old Java 8 source and target configuration.

A complete POM can look like this:

<project xmlns="https://maven.apache.org/POM/4.0.0"
         xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="
             https://maven.apache.org/POM/4.0.0
             https://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>com.mcnz.scraper</groupId>
    <artifactId>htmlunit-java25-scraper</artifactId>
    <version>1.0.0</version>

    <properties>
        <maven.compiler.release>25</maven.compiler.release>
        <project.build.sourceEncoding>
            UTF-8
        </project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.htmlunit</groupId>
            <artifactId>htmlunit</artifactId>
            <version>5.3.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.15.0</version>
            </plugin>
        </plugins>
    </build>
</project>

The old tutorial targeted Java 8 and HtmlUnit 2.34.1. Neither configuration belongs in a new Java 25 project.

Modern HtmlUnit imports

The package names changed along with modern HtmlUnit releases. The scraper uses these imports:

import org.htmlunit.WebClient;
import org.htmlunit.html.DomNode;
import org.htmlunit.html.HtmlPage;

Notice that the old com.gargoylesoftware.htmlunit package is gone.

A minimal Java 25 HtmlUnit scraper

Java 25 supports compact source files and instance main methods, so a small scraper does not need an explicit class declaration.

import org.htmlunit.WebClient;
import org.htmlunit.html.HtmlPage;

void main() throws Exception {
    var url = "https://www.theserverside.com/";

    try (var webClient = new WebClient()) {
        HtmlPage page = webClient.getPage(url);
        IO.println(page.getTitleText());
    }
}

There are several modern Java idioms in this example:

• var removes unnecessary local type repetition.
• IO.println() replaces System.out.println() for simple Java 25 console output.
• A compact void main() removes boilerplate from a small program.
• Try-with-resources closes the WebClient automatically.

WebClient implements AutoCloseable, so it belongs in a try-with-resources statement rather than being left open.

Do not disable SSL validation

The old scraper included this configuration:

webClient.getOptions().setUseInsecureSSL(true);

Do not use that as a normal scraper setting. It weakens HTTPS certificate validation and can hide real TLS configuration problems.

A modern scraper should use normal HTTPS validation unless it is deliberately interacting with a controlled development system that uses a certificate you explicitly understand and trust.

Should CSS and JavaScript be disabled?

HtmlUnit is more than an HTML parser. It simulates browser behavior and includes JavaScript support. Turning JavaScript off can therefore change the DOM you receive.

If the page is server-rendered and you only need its initial HTML, disabling CSS and JavaScript can reduce unnecessary processing:

try (var webClient = new WebClient()) {
    webClient.getOptions().setCssEnabled(false);
    webClient.getOptions().setJavaScriptEnabled(false);

    HtmlPage page = webClient.getPage(url);
}

Do this only when the content you need does not depend on JavaScript. For a client-rendered application, leave JavaScript enabled so HtmlUnit can execute the page's scripts.

Extract the HTML page title

Once WebClient.getPage() returns an HtmlPage, the title is available through getTitleText():

HtmlPage page = webClient.getPage(url);
IO.println(page.getTitleText());

The original article contained a malformed method name, getTitleText with HTML formatting inserted inside the identifier. The corrected method call is simply getTitleText().

Query the DOM with a CSS selector

HtmlUnit's DOM nodes support CSS selector queries. Suppose the author link on a page matches this selector:

#author > div > a

Query it like this:

DomNode author = page.querySelector(
    "#author > div > a"
);

if (author != null) {
    IO.println(
        "Author: " + author.asNormalizedText()
    );
}

querySelector() returns the first matching node, or null when nothing matches. A robust scraper should therefore perform a null check rather than assume the page always has the expected structure.

Extract every anchor link

HtmlPage.getAnchors() returns the page's anchor elements. Each HtmlAnchor provides getHrefAttribute() for the value of its href attribute.

for (var anchor : page.getAnchors()) {
    IO.println(anchor.getHrefAttribute());
}

This is cleaner than calling the generic getAttribute("href") method when you already know you are working with an HtmlAnchor.

Print both link text and URL

A scraper is often more useful when it records both the text visible to the user and the destination URL.

for (var anchor : page.getAnchors()) {
    var text = anchor.asNormalizedText();
    var href = anchor.getHrefAttribute();

    IO.println(
        "%s -> %s".formatted(text, href)
    );
}

Java 25's IO.println() combines nicely with String.formatted() when output contains multiple values.

Resolve relative links

An anchor might contain a relative URL such as:

<a href="/tutorial/java">Java tutorials</a>

If you need the absolute URL, ask the page to resolve the relative value against its base URL:

for (var anchor : page.getAnchors()) {
    var href = anchor.getHrefAttribute();

    if (!href.isBlank()) {
        var absoluteUrl =
            page.getFullyQualifiedUrl(href);

        IO.println(absoluteUrl);
    }
}

This is generally more useful for a crawler or content aggregator than printing raw relative links.

Complete Java 25 HtmlUnit scraper

The complete example brings the pieces together:

import org.htmlunit.WebClient;
import org.htmlunit.html.DomNode;
import org.htmlunit.html.HtmlPage;

void main() throws Exception {
    var url = "https://www.theserverside.com/";

    try (var webClient = new WebClient()) {
        webClient.getOptions()
            .setCssEnabled(false);

        HtmlPage page = webClient.getPage(url);

        IO.println(
            "Title: " + page.getTitleText()
        );

        DomNode author = page.querySelector(
            "#author > div > a"
        );

        if (author != null) {
            IO.println(
                "Author: " +
                author.asNormalizedText()
            );
        }

        IO.println();
        IO.println("Links:");

        for (var anchor : page.getAnchors()) {
            var text =
                anchor.asNormalizedText();
            var href =
                anchor.getHrefAttribute();

            if (href.isBlank()) {
                continue;
            }

            var absoluteUrl =
                page.getFullyQualifiedUrl(href);

            IO.println(
                "%s -> %s".formatted(
                    text,
                    absoluteUrl
                )
            );
        }
    }
}

This example leaves JavaScript enabled. If the target page is fully server-rendered and JavaScript is unnecessary, add:

webClient.getOptions()
    .setJavaScriptEnabled(false);

Wait for asynchronous JavaScript

If the page uses JavaScript to load content asynchronously, loading the page and immediately querying the DOM may be too early.

HtmlUnit provides mechanisms for working with its JavaScript engine, but scraper logic should wait only when the target page actually requires asynchronous client-side rendering. Avoid adding arbitrary delays to every request.

For server-rendered pages, keeping the scraper synchronous and simple is both faster and easier to maintain.

HtmlUnit vs. JSoup

HtmlUnit and JSoup overlap, but they solve somewhat different problems.

If you only need to download static HTML and extract elements, JSoup is often the simpler tool. If the workflow needs browser-like behavior, cookies, form interaction or JavaScript, HtmlUnit offers capabilities a simple HTML parser does not.

Web scraping best practices

A production scraper should do more than successfully parse HTML. It should also behave responsibly.

• Scrape only content you are authorized to access.
• Respect site terms, robots directives and applicable policies.
• Avoid sending requests at a rate that burdens the server.
• Expect CSS selectors and page structure to change.
• Handle missing elements and HTTP failures gracefully.
• Do not disable TLS verification as a shortcut.
• Close every WebClient with try-with-resources.

What changed from old HtmlUnit examples?

HtmlUnit remains a useful Java headless-browser library, but the API coordinates, package names and Java coding style have changed significantly since the old 2.x examples. With HtmlUnit 5.3.0 and Java 25, the result is cleaner Maven configuration, safer resource management and much less boilerplate.