miércoles, 13 de marzo de 2019

Selenium (2): Wait until manual selection of the certificate

0. Introduction


As we said in a previous entry in this blog, we are considering automatically use Selenium for the automatic use of web pages.

The only drawback is to manually select the certificate for authentication as it seems rather difficult for Selenium.

As Anuja Jain says in her blog there are mainly 3 ways to wait for a condition to get fulfilled.

1. First wait condition: Implicit wait

Anuja explains: "Implicit Wait means informing selenium web driver to wait for specific amount of time and if the web element is not visible after waiting for that specific point then throw "No such Element found Exception".


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
package test;

import java.util.concurrent.TimeUnit;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

public class ImplicitWait {
    public static void main(String[] args) throws Exception {
        //1. Get WebDriver
        String chromeDriverPath = "/home/eduard/Selenium/chromedriver" ;
        System.setProperty("webdriver.chrome.driver", chromeDriverPath);
        ChromeOptions options = new ChromeOptions();
        //options.addArguments("--headless", "--disable-gpu", "--window-size=1920,1200","--ignore-certificate-errors", "--silent");
        WebDriver driver = new ChromeDriver(options);
       
        driver.get("https://dantesquehtmlunit.blogspot.com");
        driver.manage().timeouts().implicitlyWait( 2000, TimeUnit.MILLISECONDS);
        driver.quit();
    }
}


2. Second wait condition: Implicit wait

In this case, Selenium is waiting until the condition is fulfilled within a max time waiting before throwing an "ElementNotVisibleException" (if the element is not found)


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
package test;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class ExplicitWait {
    public static void main(String[] args) throws Exception {
    //1. Get WebDriver
    String chromeDriverPath = "/home/eduard/Selenium/chromedriver" ;
    System.setProperty("webdriver.chrome.driver", chromeDriverPath);
    ChromeOptions options = new ChromeOptions();
    //options.addArguments("--headless", "--disable-gpu", "--window-size=1920,1200","--ignore-certificate-errors", "--silent");
    WebDriver driver = new ChromeDriver(options);
  
    //Define a wait object with a max time  for waiting
    WebDriverWait wait = new WebDriverWait(driver,20);
       
    driver.get("https://dantesquehtmlunit.blogspot.com");
  
    WebElement title;
    //Wait untill title is available
    title= wait.until(ExpectedConditions.visibilityOfElementLocated(By.className("title")));      
  
    System.out.println("condition fulfilled!");
    Thread.sleep(2000);
    driver.quit();
  }
}

A new object is created for managing the intelligent wait.

3. Third wait condition: Fluent wait. Valid for waiting to manually select Certificate for login.

By means of the fluent wait, two times are settled:

  1. First delay time
  2. repeat time cycle.
The cycle is repeated until the condition fulfils.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
package test;

import java.time.Duration;
import java.util.function.Function;

import org.openqa.selenium.By;
import org.openqa.selenium.NoSuchElementException;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.FluentWait;
import org.openqa.selenium.support.ui.Wait;

public class SeleniumWaitForCertSelection {
 
    private static WebElement getWebElement(WebDriver driver,  Wait<WebDriver> wait, String xpathExpression) {
        WebElement myElement= wait.until(new Function<WebDriver, WebElement>() {       
     public WebElement apply(WebDriver driver) { 
         return driver.findElement(By.xpath(xpathExpression));     
     }  
 });
  
 return myElement;
    }
 
    public static void main(String[] args) {
  
        //1. Get WebDriver
 String chromeDriverPath = "/home/eduard/Selenium/chromedriver" ;
 System.setProperty("webdriver.chrome.driver", chromeDriverPath);
 ChromeOptions options = new ChromeOptions();
 //options.addArguments("--headless", "--disable-gpu", "--window-size=1920,1200","--ignore-certificate-errors", "--silent");
 WebDriver driver = new ChromeDriver(options);
     
 Wait<WebDriver> wait = new FluentWait<WebDriver>(driver)       
     .withTimeout(Duration.ofSeconds(2))
     .pollingEvery(Duration.ofSeconds(1))    
     .ignoring(NoSuchElementException.class); 


 //1. Get the page
 driver.get("https://enguera.sedelectronica.es/carpetaelectronica.4");
      
   
 //2. fluent Wait and Look for the anchor element and click   (//a=anchor)
 //driver.findElement(By.xpath("//a[contains (@id, 'id2')]")).click();
 getWebElement(driver,wait,"//a[contains (@id, 'id2')]").click();


        // wait and get the second element
        getWebElement(driver,wait,"//a[contains (@id, 'id3')]").click();

        //Wait and the the nth element and meanwhile you can interact manually with the browser to get the dertificate
        getWebElement(driver,wait,"//a[contains (@id, 'idn')]").click();

        System.out.println("condition fulfilled!");
 Thread.sleep(2000);
 driver.quit();  
    }
}




Lines 16-25 a method for waiting and finding the desired element is created with yellow background

Lines 36-39 a fluent wait object is defined to manage delays

Line 46-48 explicit call to the method to find the desired element and click it,

Lines 53-54 You can interact manually with the browser until the desired element is visible. In this case, we can select manually a certificate before entering the next page






Sikulix(1): Installation for Java

0. Introduction

Sikuli seems to be an option for using certificate authentication in browsers due to the fact that the certificate selection is an OS task (see Stackoverflow) and not a browser specific action. So Selenium can hardly be useful alone for doing all tasks automatically.

Perhaps, a better solution can be select "by hand" the required certificate and tell selenium to wait until the desired page is loaded (the subsequent page after the certificate selection). See StackOverflow.



1. Installation

Reading Sikuli's docs, related to Ubuntu, it is advisable to use Ubuntu 18.04 (last releases) for avoiding problems with OpenCV installation.

I have not been able to install OpenCV version 3.2 on Ubuntu 16.04 so I have upgraded to 18.04.

These are the main prerequisites.


1
2
3
4
5
6
7
8
#1. Install libcanberra
sudo apt install libcanberra-gtk0 libcanberra-gtk-module

#2Install libopencv3.2-java
sudo apt install libopencv3.2-java

#3 Make a link with a known name for Sikuli
sudo ln -s /usr/lib/jni/libopencv_java320.so /usr/lib/libopencv_java.so

It is important to install Tesseract (version 4.1.1). Bharath Sivakumar explains how to do this task.  He also explains how to use OCR programmatically!. It is very good!

Basically (Please follow the last link!):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#1. Check Tesseract version
tesseract --version

#2. Install prerequisite "leptonica2
sudo apt-get install -y libleptonica-dev

#3.Install dependencies
sudo apt-get update -y
sudo apt-get install automake
sudo apt-get install -y pkg-config
sudo apt-get install -y libsdl-pango-dev
sudo apt-get install -y libicu-dev
sudo apt-get install -y libcairo2-dev
sudo apt-get install bc

#get the source code and unzip it
wget https://github.com/tesseract-ocr/tesseract/archive/4.1.1.zip
unzip 4.1.1.zip

#go to tesseract-4.1.1 directory and execute
./autogen.sh
./configure
make
sudo make install
sudo ldconfig
make training
sudo make training-install

#test tesseract version
tesseract --version



2. Updating the pom.xml from the last project



  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>ximodante</groupId>
  <artifactId>HTMLFillForms</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>HTMLFillForms</name>
  <description>HTMLFillForms</description>
  
  <repositories>
    <repository>
      <!-- See https://github.com/RaiMan/SikuliX1 -->
      <id>repo-sikuli</id>
      <name>sonatype nightky bluids for sikuli</name>
      <url>http://oss.sonatype.org/content/groups/public</url>
    </repository>
  </repositories>
  
  <properties>
    <cxf.version>3.2.6</cxf.version>
    <jax.version>2.3.0</jax.version>
    <lombok.version>1.18.4</lombok.version>
    <jackson.version>2.9.6</jackson.version>
    <h2.version>1.4.196</h2.version>
    <hibernate.version>5.2.16.Final</hibernate.version>
    <selenium.version>3.141.59</selenium.version>
    <poi.version>4.0.1</poi.version>
    <sikulix.version>2.0.4</sikuli.version>
    
        
    <failOnMissingWebXml>false</failOnMissingWebXml>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <!-- Java version-->
    <java.version>13</java.version>
  </properties>  
  
  <dependencies>
  
    <!-- https://mvnrepository.com/artifact/org.projectlombok/lombok -->
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <version>${lombok.version}</version>
      <scope>provided</scope>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-lang3</artifactId>
      <version>3.8.1</version>
    </dependency>
    

    <!--  HTML Parser  -->
    <!--  See https://www.mkyong.com/java/how-to-automate-login-a-website-java-example/ -->
    <!-- https://mvnrepository.com/artifact/org.jsoup/jsoup -->
    <!-- 
    <dependency>
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.11.3</version>
    </dependency>
     -->

    <!-- Simulate a browser to fill forms !!! -->
    <!--  seehttps://stackoverflow.com/a/1383097/7704658   -->
    <!-- https://mvnrepository.com/artifact/net.sourceforge.htmlunit/htmlunit -->
    <!-- 
    <dependency>
      <groupId>net.sourceforge.htmlunit</groupId>
      <artifactId>htmlunit</artifactId>
      <version>2.33</version>
    </dependency>
     -->
     
    <!-- Seleniun -->
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-java</artifactId>
      <version>${selenium.version}</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-server -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-server</artifactId>
      <version>${selenium.version}</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-firefox-driver -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-firefox-driver</artifactId>
      <version>${selenium.version}</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-chrome-driver -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-chrome-driver</artifactId>
      <version>${selenium.version}</version>
    </dependency>
    
    
    
    <!-- Sikuli ???? -->
    <dependency>
     <groupId>com.sikulix</groupId>
     <artifactId>sikulixapi</artifactId>
     <version>${sikulix.version}</version>
    </dependency>
    
    <!--  Tess4J A Java JNA wrapper for Tesseract OCR API.  -->
    <!-- https://mvnrepository.com/artifact/net.sourceforge.tess4j/tess4j -->
    <!-- 
    <dependency>
      <groupId>net.sourceforge.tess4j</groupId>
      <artifactId>tess4j</artifactId>
      <version>4.3.1</version>
    </dependency>
    -->
    
    <!-- https://mvnrepository.com/artifact/org.bytedeco.javacpp-presets/opencv -->
    <!-- 
    <dependency>
      <groupId>org.bytedeco.javacpp-presets</groupId>
      <artifactId>opencv</artifactId>
      <version>4.0.1-1.4.4</version>
    </dependency>
    -->
  
    <!--  JPA  -->
    <!-- MS-SQL Server JDBC JTDS bo de 2013-->
    <!-- https://mvnrepository.com/artifact/net.sourceforge.jtds/jtds -->
    <dependency>
      <groupId>net.sourceforge.jtds</groupId>
      <artifactId>jtds</artifactId>
      <version>1.3.1</version>
    </dependency>
     
       
    
    <!-- JPA 2.1 Provider -->
    <!-- https://mvnrepository.com/artifact/org.hibernate/hibernate-core -->
    <dependency>
      <groupId>org.hibernate</groupId>
      <artifactId>hibernate-core</artifactId>
      <version>${hibernate.version}</version>
    </dependency>
    
    <!--  END JPA -->
    
    
    <!-- For using Excel Files -->
    <!-- https://mvnrepository.com/artifact/org.apache.poi/poi -->
    <dependency>
      <groupId>org.apache.poi</groupId>
      <artifactId>poi</artifactId>
      <version>${poi.version}</version>
    </dependency>
        
    
    <!-- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml -->
    <dependency>
      <groupId>org.apache.poi</groupId>
      <artifactId>poi-ooxml</artifactId>
      <version>${poi.version}</version>
    </dependency>
    
  
    <!--BEGIN Java 9 references to JEE  not included in JDK9-->
    <!-- see http://openjdk.java.net/jeps/320-->
    
    <!-- https://mvnrepository.com/artifact/com.sun.xml.bind/jaxb-impl -->
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-impl</artifactId>
      <version>${jax.version}</version>
    </dependency>
        
    <!-- https://mvnrepository.com/artifact/com.sun.xml.bind/jaxb-core -->
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-core</artifactId>
      <version>${jax.version}</version>
    </dependency>
    
    
    <dependency>
      <groupId>com.sun.activation</groupId>
      <artifactId>javax.activation</artifactId>
      <version>1.2.0</version>
    </dependency>
    
    <dependency>
      <groupId>com.sun.xml.ws</groupId>
      <artifactId>jaxws-ri</artifactId>
      <version>${jax.version}</version>
      <type>pom</type>
    </dependency>
        
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-ri</artifactId>
      <version>${jax.version}</version>
      <type>pom</type>
    </dependency>
    
    <!-- End of Dependencies required for JAVA 9!!!! -->
    
  </dependencies>
  
  <!-- COMPILER PLUGIN -->
  <!-- This is OK  http://crunchify.com/how-to-create-build-java-project-including-all-dependencies-using-maven-maven-resources-maven-dependency-maven-jar-plugin-tutorial/
  -->
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
          <configuration>
            <!-- release>10</release -->
            <source>10</source>
            <target>10</target>
            <showWarnings>true</showWarnings>
            <showDeprecation>true</showDeprecation>
          </configuration>
          
        </plugin>
      </plugins>
    </pluginManagement>
 
    <plugins>
      <plugin>
        <artifactId>maven-resources-plugin</artifactId>
        <version>3.1.0</version>
        <executions>
          <execution>
            <id>copy-resources</id>
            <phase>validate</phase>
            <goals>
              <goal>copy-resources</goal>
            </goals>
            <configuration>
              <outputDirectory>${basedir}/target/MyFolder</outputDirectory>
              <resources>
                <resource>
                  <directory>resources</directory>
                  <filtering>true</filtering>
                </resource>
              </resources>
            </configuration>
          </execution>
        </executions>
      </plugin>
 
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>3.1.1</version>
        <executions>
          <execution>
            <id>copy-dependencies</id>
            <phase>prepare-package</phase>
            <goals>
              <goal>copy-dependencies</goal>
            </goals>
            <configuration>
              <outputDirectory>${project.build.directory}/MyFolder/lib</outputDirectory>
              <overWriteReleases>false</overWriteReleases>
              <overWriteSnapshots>false</overWriteSnapshots>
              <overWriteIfNewer>true</overWriteIfNewer>
            </configuration>
          </execution>
        </executions>
      </plugin>
   
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <classpathPrefix>lib/</classpathPrefix>
              <mainClass>u.request.MyExecutableClass</mainClass>
            </manifest>
            <manifestEntries>
              <Class-Path>.</Class-Path>
            </manifestEntries>
          </archive>
 
          <finalName>MyFolder/MyJars</finalName>
        </configuration>
      </plugin>
    </plugins>
  
  </build>
  
</project>
  


The new changes are displayed on a yellow background:

Lines 9-16 indicates the new maven repository to find the latest Sikuli version as states in the "README.md" file in the  Sikuli github.

Line 27 indicates the Sikulix version

Lines 107-112 indicates the dependency to add to the project.


3. The easy example

This is a similar example to all tutorials!!!!
  1. Right-click on the desktop window and create a new folder and name it "TestFolder".
  2. Use Shutter screenshot for selecting the image of the folder. As this one
  3. Name this image as "TestFolderImage.png" 
  4. Put it in a folder like "/home/user/myImages"  (where "user" is your username!!!)
  5. This is the address of the patter to find in the desktop screen "/home/user/myImages/TestFolderImage.png"
  6. Let's create a Screen element and a Pattern element and let's double click on the part of the Desktop region that matches this pattern image.
Here is the code 


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
package test;

import org.sikuli.script.FindFailed;
import org.sikuli.script.Pattern;
import org.sikuli.script.Screen;

public class Sikuli02 {

  public static void main(String[] args) throws FindFailed {
    Screen s = new Screen();
    Pattern folderToFind = new Pattern("/home/ximo/myImages/TestFolderImage.png");
    // Finds the image on the desktop and doubleclick on it
    s.doubleClick(folderToFind);  
  }
}





domingo, 10 de marzo de 2019

Selenium(1): Selenium: First Project

0. Introduction


In Edureka there are plenty of instructions for installing Selenium. This web has been my inspiration source.

The main requirements are:

  1. Java SDK (version 10 or newer version) from Oracle. Note that the link points to JDK 11 and I am using java 10!!! in this post
  2. Elipse IDE (Photon or newer version)
  3. Project Lombok (only if you want to avoid Setters and Getters verbosity)
  4. Selenium Java, or maybe better, this link for ChromeDriver 
  5. Adapted executable browsers.

1. Installing adapted browsers

Selenium need adapted browser versions that should be downloaded. You should install them and write down the route to these executables. On the Selenium download page are the links to these adapted browsers:





UPDATE 2020.12.16: You can download chrome driver from https://chromedriver.chromium.org/downloads

2. The maven pom.xml file

You should create a maven project as we did in the first post of this blog. Let's see the dependencies in the file:


  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>ximodante</groupId>
  <artifactId>HTMLFillForms</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>HTMLFillForms</name>
  <description>HTMLFillForms</description>
  
  
  <properties>
    <cxf.version>3.2.6</cxf.version>
    <jax.version>2.3.0</jax.version>
    <lombok.version>1.18.4</lombok.version>
    <jackson.version>2.9.6</jackson.version>
    <h2.version>1.4.196</h2.version>
    <hibernate.version>5.2.16.Final</hibernate.version>
    
        
    <failOnMissingWebXml>false</failOnMissingWebXml>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <!-- Java version-->
    <java.version>10</java.version>
  </properties>  
  
  <dependencies>
  
    <!-- https://mvnrepository.com/artifact/org.projectlombok/lombok -->
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <version>${lombok.version}</version>
      <scope>provided</scope>
    </dependency>
    

     

    <!-- Simulate a browser to fill forms !!! -->
      <!-- Seleniun -->
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-java</artifactId>
      <version>3.141.59</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-server -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-server</artifactId>
      <version>3.141.59</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-firefox-driver -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-firefox-driver</artifactId>
      <version>3.141.59</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-chrome-driver -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-chrome-driver</artifactId>
      <version>3.141.59</version>
    </dependency>
    
    
    
  
    <!--  JPA  -->
    <!-- MS-SQL Server JDBC JTDS bo de 2013-->
    <!-- https://mvnrepository.com/artifact/net.sourceforge.jtds/jtds -->
    <dependency>
      <groupId>net.sourceforge.jtds</groupId>
      <artifactId>jtds</artifactId>
      <version>1.3.1</version>
    </dependency>
     
       
    
    <!-- JPA 2.1 Provider -->
    <!-- https://mvnrepository.com/artifact/org.hibernate/hibernate-core -->
    <dependency>
      <groupId>org.hibernate</groupId>
      <artifactId>hibernate-core</artifactId>
      <version>${hibernate.version}</version>
    </dependency>
    
    <!--  END JPA -->
  
    <!--BEGIN Java 9 references to JEE  not included in JDK9-->
    <!-- see http://openjdk.java.net/jeps/320-->
    
    <!-- https://mvnrepository.com/artifact/com.sun.xml.bind/jaxb-impl -->
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-impl</artifactId>
      <version>${jax.version}</version>
    </dependency>
        
    <!-- https://mvnrepository.com/artifact/com.sun.xml.bind/jaxb-core -->
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-core</artifactId>
      <version>${jax.version}</version>
    </dependency>
    
    
    <dependency>
      <groupId>com.sun.activation</groupId>
      <artifactId>javax.activation</artifactId>
      <version>1.2.0</version>
    </dependency>
    
    <dependency>
      <groupId>com.sun.xml.ws</groupId>
      <artifactId>jaxws-ri</artifactId>
      <version>${jax.version}</version>
      <type>pom</type>
    </dependency>
        
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-ri</artifactId>
      <version>${jax.version}</version>
      <type>pom</type>
    </dependency>
    
    <!-- End of Dependencies required for JAVA 9!!!! -->
    
  </dependencies>
  
  <!-- COMPILER PLUGIN -->
  <!-- This is OK  http://crunchify.com/how-to-create-build-java-project-including-all-dependencies-using-maven-maven-resources-maven-dependency-maven-jar-plugin-tutorial/
  -->
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
          <configuration>
            <!-- release>10</release -->
            <source>10</source>
            <target>10</target>
            <showWarnings>true</showWarnings>
            <showDeprecation>true</showDeprecation>
          </configuration>
          
        </plugin>
      </plugins>
    </pluginManagement>
 
    <plugins>
      <plugin>
        <artifactId>maven-resources-plugin</artifactId>
        <version>3.1.0</version>
        <executions>
          <execution>
            <id>copy-resources</id>
            <phase>validate</phase>
            <goals>
              <goal>copy-resources</goal>
            </goals>
            <configuration>
              <outputDirectory>${basedir}/target/MyFolder</outputDirectory>
              <resources>
                <resource>
                  <directory>resources</directory>
                  <filtering>true</filtering>
                </resource>
              </resources>
            </configuration>
          </execution>
        </executions>
      </plugin>
 
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>3.1.1</version>
        <executions>
          <execution>
            <id>copy-dependencies</id>
            <phase>prepare-package</phase>
            <goals>
              <goal>copy-dependencies</goal>
            </goals>
            <configuration>
              <outputDirectory>${project.build.directory}/MyFolder/lib</outputDirectory>
              <overWriteReleases>false</overWriteReleases>
              <overWriteSnapshots>false</overWriteSnapshots>
              <overWriteIfNewer>true</overWriteIfNewer>
            </configuration>
          </execution>
        </executions>
      </plugin>
   
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <classpathPrefix>lib/</classpathPrefix>
              <mainClass>u.request.MyExecutableClass</mainClass>
            </manifest>
            <manifestEntries>
              <Class-Path>.</Class-Path>
            </manifestEntries>
          </archive>
 
          <finalName>MyFolder/MyJars</finalName>
        </configuration>
      </plugin>
    </plugins>
  
  </build>
  
</project>
  


Lines 40 to 66 indicates the selenium dependencies (server, java and optionally chrome and firefox).

Lines 27 to 33 show Lombok dependency

Lines 71 to 90 show JPA dependencies (For MS-SQL server and Hibernate)

Lines 92 to 130 add XML dependencies and other stuff not included in Java 9 and newer versions.

Lines 134 and next uses a compiler plugin for Java10

REMARKS: I have used Java 10. If you are using other Java version, you should arrange all the lines that make reference to Java version 10 !!!

3. A simple class

This class :

  1. Opens a Chrome browser 
  2. Fills user and password 
  3. Submits the form.
  4. Closes the browser


You must know the route you installed the adapted browser!! In my case it is in "/home/ximodante/Selenium/chromedriver"


String chromeDriverPath = "/home/ximodante/Selenium/chromedriver" ;

System.setProperty("webdriver.chrome.driver", chromeDriverPath);

Xpath (version 1.0 !! that is very limited compared with version 2) is used in Chrome to find the component to fill, and the sendKeys() method is used to fill the form. To press a button, the click() method is used.

Here is the code:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
package u.requests;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;


/**
 * @see https://www.edureka.co/blog/selenium-installation/
 * @author ximo dante
 *
 */
public class SeleniumTest {

 public static void main(String[] args) {
  
     //1. Get WebDriver
     String chromeDriverPath = "/home/eduard/Selenium/chromedriver" ;
     System.setProperty("webdriver.chrome.driver", chromeDriverPath);
     ChromeOptions options = new ChromeOptions();
     //options.addArguments("--headless", "--disable-gpu", "--window-size=1920,1200","--ignore-certificate-errors", "--silent");
     WebDriver driver = new ChromeDriver(options);


     driver.get("https://contrataciondelestado.es/wps/portal/organismosPublicos");
     driver.findElement(By.xpath("//input[contains (@id, 'input_userID')]")).sendKeys("myUser");
     driver.findElement(By.xpath("//input[contains (@id,'input_password')]")).sendKeys("myPassword");
     driver.findElement(By.xpath("//input[contains (@name, '_login')]")).click();
  
     System.out.println(driver.getPageSource());
  
     driver.close();
 }

}

Happy coding


martes, 5 de marzo de 2019

HtmlUnit(4): Https logging with client certificate

I have not tested this case, so I beg your pardon for this post! but here is good documentation.


See Roshith response in stackoverflow.


WebClientOptions.setSSLClientCertificate(URL certificateUrl,String certificatePassword, String certificateType)

To this method can be accessed using a stream (containing the certificate)




To get a card certificate, see also in stackoverflow:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
    String pkcs11Config = "name = SmartCard\nlibrary = /path/to/libraby.so";
    ByteArrayInputStream confStream = new ByteArrayInputStream(pkcs11Config.getBytes());
    Provider prov = new sun.security.pkcs11.SunPKCS11(confStream);
    Security.addProvider(prov);
    KeyStore cc = null;
    String pin = "";
    try {
        cc = KeyStore.getInstance("PKCS11",prov);
        KeyStore.PasswordProtection pp = new KeyStore.PasswordProtection(pin.toCharArray());
        cc.load(null ,  pp.getPassword() );
        Enumeration aliases = cc.aliases();
        while (aliases.hasMoreElements()) {
            Object alias = aliases.nextElement();
            try {
                X509Certificate cert0 = (X509Certificate) cc.getCertificate(alias.toString());
                System.out.println("I am: " + cert0.getSubjectDN().getName());
            } catch (Exception e) {
                continue;
            }
        }
    } catch (Exception e) {
        e.printStackTrace();
    }


lunes, 4 de marzo de 2019

Htmlunit(3): HtmlUnit known problems

Here is a list of possible problems:

1. How to fill a "select field"?

Here is an example


HtmlSelect mySelect = ...

// 1. Setting the value directly 
HtmlPage myPage = mySelect.setSelectedAttribute("1" , true);

// 2. OR using an Option object
HtmlOption option = mySelecty.getOptionByValue("1");
myPage=mySelect.setSelectedAttribute(option, true);
      /

2. How to declare the use of  Ajax?

Here is a method for creating a WebClient with Ajax.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
private static WebClient getWebClient(boolean useInsecureSSL) {
  WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED);
  
  // @see https://stackoverflow.com/a/28543031/7704658
  // Avoid certificate problems
  webClient.getOptions().setUseInsecureSSL(useInsecureSSL);
  
  //Use Ajax
  webClient.setAjaxController(new NicelyResynchronizingAjaxController());

  //Use Javascript
  webClient.getOptions().setJavaScriptEnabled(true);

  webClient.getOptions().setActiveXNative(false);

  webClient.getOptions().setCssEnabled(true);

  webClient.getOptions().setThrowExceptionOnScriptError(true);
  
  webClient.getCookieManager().setCookiesEnabled(true);


  return webClient;
 }
 


3. How can we realise that we are trying to access a field that does not exist?

There is javascript runtime error complaining about cannot access some attribute (style etc) from "null"

4. Waiting for events like "onchange" ?

First, you should use a WebClient as the one in the first question. Then, after filling the field info, you should use something like this (wait for during some time interval)


1
2
3
4
5
6
7
8
//Update the select field to option "1"
myPage=myFld.setSelectedAttribute("1", true);

//if exists an onchange method
if (myFld.getOnChangeAttribute().length()>1) {
    webClient.waitForBackgroundJavaScript(20000);
    myPag = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();
}



5. If an event like "onchange" tries to access a field to change the visibility (style) and cannot find this field?

Some times we are trying to connect to a page that may have problems. It happened to me when a javascript function in the "onchange" attribute of a select, tried to access a field by an id and could not find it.
I had to create a field in the form with this id previous to update the field that had the onchange action.

Here is the code of adding an HtmlElement with type "div" and  id="divAcuerdoMarcoSeleccionado


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
/**
  * @see https://stackoverflow.com/a/6172374/7704658 for adding elements
  * @param elem
  */
 private static void treatPossibleErrors(HtmlElement elem, HtmlPage htmlPage) {
  // Page04 el select _:form1:tipo al cambiar busca un elemento que no encuentra
  if (elem.getClass().getSimpleName().equalsIgnoreCase("HtmlSelect") &&
   elem.getAttribute("name").toLowerCase().endsWith("_:form1:tipo") && 
   elem.getAttribute("onchange").contains("seleccionarDerivadoAcuerdoMarco(this, 7,3)")) {
   
   HtmlElement createdMockElement = (HtmlElement) htmlPage.createElement("input");
   createdMockElement.setAttribute("type", "div");
   createdMockElement.setAttribute("id", "divAcuerdoMarcoSeleccionado");
   createdMockElement.setAttribute("name", "divAcuerdoMarcoSeleccionado");

   HtmlForm form=elem.getEnclosingForm();
   form.appendChild(createdMockElement);
  }
 }


6. The worst error: Javascript incompatibility

This error has made me change the scrapper and abandon "Htmlunit". After filling all data and spending many hours, in the last screen, the "accept data" button raised a "click exception". I tried to analyse the javascript code, but it was not successful. So I have tried Selenium.