WebDriver, is a fantastic Java API for web application testing. It has recently been merged into the Selenium project to provide a friendlier API for programmatic simulation of web browser actions. Its unique property is that of executing web pages on web browsers such as Firefox, Chrome, IE etc, and the subsequent programmatic access of the DOM model.

The problem with WebDriver, though, as reported here, is that because the underlying browser implementation does the actual fetching, as opposed to, Commons HttpClient, for example, its currently not possible to obtain the HTTP request and response headers, which is kind of a PITA.

I present here a method of obtaining HTTP request and response headers via an embedded proxy, derived from the Proxoid project.

ProxyLight from Proxoid

ProxyLight is the lightweight standalone proxy from the Proxoid project. It's released under the Apache Public License.

The original code only provided request filtering, and performed no response filtering, forwarding data directly from the web server to the requesting client.

I made some modifications to intercept and parse HTTP response headers.

Get my version here (released under APL): http://downloads.supermind.org/proxylight-20110622.zip

Using ProxyLight from WebDriver

The modified ProxyLight allows you to process both request and response.

This has the added benefit allowing you to write a RequestFilter which ignores images, or URLs from certain domains. Sweet!

What your WebDriver code has to do then, is:

  1. Ensure the ProxyLight server is started
  2. Add Request and Response Filters to the ProxyLight server
  3. Maintain a cache of request and response filters which you can then retrieve
  4. Ensure the native browser uses our ProxyLight server

Here's a sample class to get you started

span class="co2">import com.mba.proxylight.ProxyLight;
import com.mba.proxylight.Response;
import com.mba.proxylight.ResponseFilter;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;

import java.util.LinkedHashMap;
import java.util.Map;
// LRU response table. Note: this is not thread-safe.
  // Use ConcurrentLinkedHashMap instead: http://code.google.com/p/concurrentlinkedhashmap/
/**
     * Get the native browser to use our proxy
     */
"network.proxy.type""network.proxy.http", "localhost""network.proxy.http_port"// Now fetch the URL
// this response filter adds the intercepted response to the cache
// add request filters here if needed

    // now start the proxy
"http://www.lucenetutorial.com"

  • http://edgeofvision.com Neil H.

    Thanks for posting this! I've been able to modify this code to work using FirefoxDriver and DesiredCapabilities, but no such luck with RemoteWebDriver and DesiredCapabilities. Have you had any success with the latter? Let me know if you'd like me to post what I've tried so far.

    I'm pretty sure this is an issue having to do with RemoteWebDriver and proxies in general, although unfortunately documentation on this seems to be pretty scant.

  • Kelvin

    Hey Neil, I haven't tried getting RemoteWebDriver to work myself, but here are some (obvious) suggestions:

    1. Ensure the proxy host is not "localhost" but the hostname of your proxy
    2. Get Selenium server to use the proxy: http://wiki.openqa.org/display/SRC/Proxy+Configuration+for+the+Selenium+Server

    Let me know how you go?

  • http://www.pubbitch.org/blog Simon Stewart

    The recommended way to do this is using the org.openqa.selenium.Proxy class:

    Proxy proxy = new Proxy();
    proxy.setHttpProxy("localhost:" + proxyPort);

    DesiredCapabilities caps = DesiredCapabilities.firefox();
    caps.setCapability(CapabilityType.PROXY, proxy);

    new FirefoxDriver(caps);

  • Krishnan Mahadevan

    Simon,
    I have been having a tough time trying to get RemoteWebDriver to work with IE especially when trying to set the proxy settings. Here's what I have:

    Proxy proxy = new Proxy();
    proxy.setProxyAutoconfigUrl(MYPROXY_PAC_URL_GOES_HERE);
    DesiredCapabilities capability = new DesiredCapabilities();
    capability.setBrowserName(DesiredCapabilities.internetExplorer().getBrowserName());
    capability.setCapability(CapabilityType.PROXY, proxy);
    RemoteWebDriver d = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capability);
    Selenium s = new WebDriverBackedSelenium(d, "http://www.google.com");

    The line
    RemoteWebDriver d = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capability);

    keeps giving me this error:
    org.openqa.selenium.WebDriverException: Proxy autodetect is incompatible with manual settings
    Build info: version: '2.0rc3', revision: 'unknown', time: '2011-06-29 11:21:17'
    System info: os.name: 'Windows XP', os.arch: 'x86', os.version: '5.1', java.version: '1.6.0_20'
    Driver info: driver.version: RemoteWebDriver

    Following is how I am starting my local hub and a local webdriver node

    start java -jar selenium-server-standalone-2.0rc3.jar -role hub
    start java -jar selenium-server-standalone-2.0rc3.jar -role webdriver -hub http://localhost:4444/grid/register -port 5556

    Any pointers as to what am I missing here ?

  • http://edgeofvision.com Neil H.

    Btw, there's now a ticket for the "Unable to set proxy via DesiredCapabilities in RemoteWebDriver" issue, as it seems to have been experienced by a number of different folks:

    http://code.google.com/p/selenium/issues/detail?id=2051

  • Krishnan Mahadevan

    Neil,

    That was me who filed the bug the same guy who posted right above your post :)
    I eventually ended up running pillar to post, before I found out that it is a good candidate for being logged in as a bug.

    Krishnan

  • Kelvin

    Neil and Krishnan, thanks for posting about the bug re: DesiredCapabilities and RemoteWebDriver.

    It clarifies things for folks who are reading this post.

  • andriyko

    is it possible to use this proxy for https traffic?

  • Pingback: A Smattering of Selenium #58 « Official Selenium Blog()

  • Nerunja

    The download link http://downloads.supermind.org/proxylight-20110622.zip
    is not working. Please, provide the right link.

  • ForAKhaliq

    My name is Abdul Khaliq, I'm having same problem and is in need of reading response headers of HTTP in selinium. if you figured out any solution kindly contact me. Thanks. Infect most of code I get relating to CaptureNetworkTraffic used in java and not getting any code snippet of php (prefereably yii framework)

    forAKhaliq@yahoo.com

  • Anonymous

    Sorry about that! I just re-uploaded the file. Please try again.

  • C Pizzle Pham

    Is there a new download link for the project? The provided link is not working.

  • Pawangaria

    HI kelvin,
    i am new to this webdriver so can you please help me with this like i am using your above code to get the response headers for my website.
    but after executing the code the website get opened in Firefox but and its not executing the syso statement for the Headers.

    can you please help me out,

    Thanks in advance.

    pawan

  • ac

    Any updates on this. Did you able to use RemoteWebDriver and proxyLight together

  • Xlzdew

    Hi, I tried your scripts and it seems work well, but I want to get the response status(200/302..etc) info for each request either, could you help me figure out?

    Thanks very much,
    xlzdew
    xlzdewgood@gmail.com

  • Gaurav

    Hi Kelvin,

    Thanks for the above proxy code.. But there are some problems coming. I am not able to setup in my project. There are some problems coming which I am not able to resolve. Please help in this..

  • Mallikarjuna Sarva

    Hi All

    This Mallikarjuna, Here we runing same above program, But following error is shows in console:

    java.net.BindException: Address already in use: bind
    at sun.nio.ch.Net.bind(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
    at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
    at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
    at com.mba.proxylight.ProxyLight$1.run(ProxyLight.java:61)
    at java.lang.Thread.run(Unknown Source)
    Exception in thread "main" java.lang.NullPointerException
    at moolya.traffic_capture.SampleWebDriver.main(SampleWebDriver.java:75)

    Please help
    Thanks

  • Pooja

    Hi Kelvin,

    Thanks for posting this, however I want to fetch the post data in request in order to validate the right parameters has been sent into the request or not, how do I do that!

  • Nabhanya Varma

    Thank you for sharing..
    I will try to execute the above program.
    http://seleniumeasy.com

  • harry

    Hi Kelvin,

    Thanks for posting this, this tool is really handy and I use it very exhaustively :) . In my case I am able to implement successfully, but with certain ajax cases, I am getting only the request headers and not able to obtain the response headers (The behavior is very erratic, sometimes I get the response headers and sometimes I don't ) . Any comments will be really helpful.

    Thanks

  • Pingback: Fix Java Error Parsing Http Request Header Windows XP, Vista, 7, 8 [Solved]()

  • http://www.testingexcellence.com Amir Ghahrai
  • http://www.testingexcellence.com Amir Ghahrai
  • mobileapps

    Nice Article visit http://www.mobileappstesting.com for getting updates on mobile apps testing, android testing

  • Stephanie Reinhardt

    Thank you for sharing article, also you can see more detail of iOS testing or mobile testing application link on http://www.gatewaytechnolabs.com/mobile-application-testing

  • http://www.techbeamers.com/ Dasagriva Manu

    Very clean code and informative material. Such info is helpful when you like to know if a page is up or in maintenance. Selenium Webdriver is indeed a powerful tool and we can scale it to meet our expectations. To learn advanced Selenium skills, just check out http://www.techbeamers.com/ if you wish to.