Htmlunit(3): HtmlUnit known problems

Here is a list of possible problems:

1. How to fill a "select field"?

Here is an example

HtmlSelect mySelect = ...

// 1. Setting the value directly 
HtmlPage myPage = mySelect.setSelectedAttribute("1" , true);

// 2. OR using an Option object
HtmlOption option = mySelecty.getOptionByValue("1");
myPage=mySelect.setSelectedAttribute(option, true);
      /

2. How to declare the use of  Ajax?

Here is a method for creating a WebClient with Ajax.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

private static WebClient getWebClient(boolean useInsecureSSL) { WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED); // @see https://stackoverflow.com/a/28543031/7704658 // Avoid certificate problems webClient.getOptions().setUseInsecureSSL(useInsecureSSL); //Use Ajax webClient.setAjaxController(new NicelyResynchronizingAjaxController()); //Use Javascript webClient.getOptions().setJavaScriptEnabled(true); webClient.getOptions().setActiveXNative(false); webClient.getOptions().setCssEnabled(true); webClient.getOptions().setThrowExceptionOnScriptError(true); webClient.getCookieManager().setCookiesEnabled(true); return webClient; }

3. How can we realise that we are trying to access a field that does not exist?

There is javascript runtime error complaining about cannot access some attribute (style etc) from "null"

4. Waiting for events like "onchange" ?

First, you should use a WebClient as the one in the first question. Then, after filling the field info, you should use something like this (wait for during some time interval)

1 2 3 4 5 6 7 8

//Update the select field to option "1" myPage=myFld.setSelectedAttribute("1", true); //if exists an onchange method if (myFld.getOnChangeAttribute().length()>1) { webClient.waitForBackgroundJavaScript(20000); myPag = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage(); }

5. If an event like "onchange" tries to access a field to change the visibility (style) and cannot find this field?

Some times we are trying to connect to a page that may have problems. It happened to me when a javascript function in the "onchange" attribute of a select, tried to access a field by an id and could not find it.
I had to create a field in the form with this id previous to update the field that had the onchange action.

Here is the code of adding an HtmlElement with type "div" and  id="divAcuerdoMarcoSeleccionado" 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

/** * @see https://stackoverflow.com/a/6172374/7704658 for adding elements * @param elem */ private static void treatPossibleErrors(HtmlElement elem, HtmlPage htmlPage) { // Page04 el select _:form1:tipo al cambiar busca un elemento que no encuentra if (elem.getClass().getSimpleName().equalsIgnoreCase("HtmlSelect") && elem.getAttribute("name").toLowerCase().endsWith("_:form1:tipo") && elem.getAttribute("onchange").contains("seleccionarDerivadoAcuerdoMarco(this, 7,3)")) { HtmlElement createdMockElement = (HtmlElement) htmlPage.createElement("input"); createdMockElement.setAttribute("type", "div"); createdMockElement.setAttribute("id", "divAcuerdoMarcoSeleccionado"); createdMockElement.setAttribute("name", "divAcuerdoMarcoSeleccionado"); HtmlForm form=elem.getEnclosingForm(); form.appendChild(createdMockElement); } }

6. The worst error: Javascript incompatibility

This error has made me change the scrapper and abandon "Htmlunit". After filling all data and spending many hours, in the last screen, the "accept data" button raised a "click exception". I tried to analyse the javascript code, but it was not successful. So I have tried Selenium.