Why Selenium Server takes screenshots on exceptions by default?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Why Selenium Server takes screenshots on exceptions by default?

Alexei Barantsev
Hi, devs,

Why Selenium Server takes screenshots on exceptions by default? It's huge performance penalty.

An example:

      for (int i = 1; i < 100; i++) {
        try {
          driver.get("test.com");
        } catch (WebDriverException ex) {
        }
      }

This piece of code takes ~25 seconds and generates ~20MB of screenshots in ChromeDriver on my machine with 1920x1200 sceen resolution using the server running on the same machine.

With capability "webdriver.remote.quietExceptions" set to true the same code takes ~1,7 seconds.

15x time perfomance boost and -20MB of useless traffic.

And again the question is: why this feature is ON by default?

Regards,
-- 
Alexei Barantsev
Software-Tesing.Ru
Selenium2.Ru

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Simon Stewart
Same reason we include all sorts of information in our exceptions: because it's proved the best way to provide debugging information that people weren't otherwise providing. 

Normally, an exception marks the end of a test, so your example is a little contrived. The only case where I've seen people doing something that might do otherwise on a regular basis is checking to see if an element is present (by using "findElement" rather than "findElements").

In real life usage, is this causing you problems?

Simon

On Fri, Dec 23, 2016 at 2:26 PM, Alexei Barantsev <[hidden email]> wrote:
Hi, devs,

Why Selenium Server takes screenshots on exceptions by default? It's huge performance penalty.

An example:

      for (int i = 1; i < 100; i++) {
        try {
          driver.get("test.com");
        } catch (WebDriverException ex) {
        }
      }

This piece of code takes ~25 seconds and generates ~20MB of screenshots in ChromeDriver on my machine with 1920x1200 sceen resolution using the server running on the same machine.

With capability "webdriver.remote.quietExceptions" set to true the same code takes ~1,7 seconds.

15x time perfomance boost and -20MB of useless traffic.

And again the question is: why this feature is ON by default?

Regards,
-- 
Alexei Barantsev
Software-Tesing.Ru
Selenium2.Ru

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYHtrD8cdj1EfjRL6ZUAK7aVJp-ZMthRwcCdGybshRHe%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Alexei Barantsev
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

4) How is it supposed to be used? Do all test frameworks (JUnit/NUnit/py.test/rspec/mocha/...) allow to examine the exception occured during test execution in a kind of @After-block? No. Should a user catch an exception in every test and examine and save the screenshot if present?

5) And it is present there *only* if it is a remote session. If you run test locally you have to implement taking screenshots on failure anyway. So they do it in local/remove agnostic manner.

My suggestion:
a) disable this feature by default, because it is not very popular, but gives performance penalty to all users,
b) may be implement it for local runs too.

Alexei Barantsev,
-- 
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 0:31:15 UTC+3 пользователь Simon Stewart написал:
Same reason we include all sorts of information in our exceptions: because it's proved the best way to provide debugging information that people weren't otherwise providing. 

Normally, an exception marks the end of a test, so your example is a little contrived. The only case where I've seen people doing something that might do otherwise on a regular basis is checking to see if an element is present (by using "findElement" rather than "findElements").

In real life usage, is this causing you problems?

Simon

On Fri, Dec 23, 2016 at 2:26 PM, Alexei Barantsev <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="HhZ8b_QrEQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">bara...@...> wrote:
Hi, devs,

Why Selenium Server takes screenshots on exceptions by default? It's huge performance penalty.

An example:

      for (int i = 1; i < 100; i++) {
        try {
          driver.get("<a href="http://test.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftest.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHISpVEBs4W2GxgRr8fjtKlrkVGeQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftest.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHISpVEBs4W2GxgRr8fjtKlrkVGeQ&#39;;return true;">test.com");
        } catch (WebDriverException ex) {
        }
      }

This piece of code takes ~25 seconds and generates ~20MB of screenshots in ChromeDriver on my machine with 1920x1200 sceen resolution using the server running on the same machine.

With capability "webdriver.remote.quietExceptions" set to true the same code takes ~1,7 seconds.

15x time perfomance boost and -20MB of useless traffic.

And again the question is: why this feature is ON by default?

Regards,
-- 
Alexei Barantsev
Software-Tesing.Ru
Selenium2.Ru

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="HhZ8b_QrEQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">selenium-developers+unsubscribe@....
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/35b7ec88-c56e-4fc0-9c48-e9a96a939959%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Paul Hammant-3
In reply to this post by Alexei Barantsev
I don't think it does take screenshots by default/silently:

ChromeDriver chromeDriver = new ChromeDriver();
long start = System.currentTimeMillis();
for (int i = 1; i < 100; i++) {
    try {
        chromeDriver.get("test.com");
    } catch (WebDriverException ex) {
    }
}
System.err.println("millis " + (System.currentTimeMillis() - start));

3.4 seconds on my 2011 Mac with 4GB RAM, no pngs the current directory.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CA%2B298UhDeK0ZB2%2B_QhBtbLQTMbq9nVaOw348hBgzwoEn2bZYnA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Alexei Barantsev
Yes, if you run locally -- there are no screenshots.
Selenium Server takes them and attach them to exceptional responses.

Regards
-- 
Alexei Barantsev
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 14:25:00 UTC+3 пользователь Paul Hammant написал:
I don't think it does take screenshots by default/silently:

ChromeDriver chromeDriver = new ChromeDriver();
long start = System.currentTimeMillis();
for (int i = 1; i < 100; i++) {
    try {
        chromeDriver.get("<a href="http://test.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftest.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHISpVEBs4W2GxgRr8fjtKlrkVGeQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftest.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHISpVEBs4W2GxgRr8fjtKlrkVGeQ&#39;;return true;">test.com");
    } catch (WebDriverException ex) {
    }
}
System.err.println("millis " + (System.currentTimeMillis() - start));

3.4 seconds on my 2011 Mac with 4GB RAM, no pngs the current directory.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/512df7c7-8dc8-488b-8899-95a003a8293d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Paul Hammant-3
Could you help me reproduce this, please :)

On Sun, Dec 25, 2016 at 2:17 PM, Alexei Barantsev <[hidden email]> wrote:
Yes, if you run locally -- there are no screenshots.
Selenium Server takes them and attach them to exceptional responses.

Regards
-- 
Alexei Barantsev
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 14:25:00 UTC+3 пользователь Paul Hammant написал:
I don't think it does take screenshots by default/silently:

ChromeDriver chromeDriver = new ChromeDriver();
long start = System.currentTimeMillis();
for (int i = 1; i < 100; i++) {
    try {
        chromeDriver.get("test.com");
    } catch (WebDriverException ex) {
    }
}
System.err.println("millis " + (System.currentTimeMillis() - start));

3.4 seconds on my 2011 Mac with 4GB RAM, no pngs the current directory.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/512df7c7-8dc8-488b-8899-95a003a8293d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CA%2B298UjCn_w_9N8QXYJthGaWfrhc-AzT1-4yh%2Bur2JS_C-EBrw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Alexei Barantsev
1) Start Selenium Server
2) Run the sample:

    DesiredCapabilities caps = DesiredCapabilities.chrome();
    //caps.setCapability("webdriver.remote.quietExceptions", true);
    WebDriver driver = new RemoteWebDriver(caps);
    long start = System.currentTimeMillis();
    for (int i = 1; i < 100; i++) {
      try {
        driver.get("test.com");
      } catch (WebDriverException ex) {
      }
    }
    System.err.println("millis " + (System.currentTimeMillis() - start));
    driver.quit();

3) Try to uncomment the line that sets "webdriver.remote.quietExceptions" capability and run again.

Regards,
-- 
Alexei Barantsev
Software-Testing.Ru
Selenium2.Ru


понедельник, 26 декабря 2016 г., 0:18:42 UTC+3 пользователь Paul Hammant написал:
Could you help me reproduce this, please :)

On Sun, Dec 25, 2016 at 2:17 PM, Alexei Barantsev <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Btr6xdl5EQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">bara...@...> wrote:
Yes, if you run locally -- there are no screenshots.
Selenium Server takes them and attach them to exceptional responses.

Regards
-- 
Alexei Barantsev
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 14:25:00 UTC+3 пользователь Paul Hammant написал:
I don't think it does take screenshots by default/silently:

ChromeDriver chromeDriver = new ChromeDriver();
long start = System.currentTimeMillis();
for (int i = 1; i < 100; i++) {
    try {
        chromeDriver.get("<a href="http://test.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftest.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHISpVEBs4W2GxgRr8fjtKlrkVGeQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftest.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHISpVEBs4W2GxgRr8fjtKlrkVGeQ&#39;;return true;">test.com");
    } catch (WebDriverException ex) {
    }
}
System.err.println("millis " + (System.currentTimeMillis() - start));

3.4 seconds on my 2011 Mac with 4GB RAM, no pngs the current directory.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="Btr6xdl5EQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">selenium-developers+unsubscribe@....
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/selenium-developers/512df7c7-8dc8-488b-8899-95a003a8293d%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/selenium-developers/512df7c7-8dc8-488b-8899-95a003a8293d%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/selenium-developers/512df7c7-8dc8-488b-8899-95a003a8293d%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/selenium-developers/512df7c7-8dc8-488b-8899-95a003a8293d%40googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/8161fa80-0645-4ae7-b14e-08683e675b84%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Simon Stewart
In reply to this post by Alexei Barantsev
After a heck a pause, replying. Inline.

On Sun, Dec 25, 2016 at 9:08 AM, Alexei Barantsev <[hidden email]> wrote:
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

Let's fix it, then. Given the discussions around IDE, we can just leave that.
 
2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

That's why we write javadocs. And why things appear in stacktraces. We try and provide as much hand-holding as makes sense, but I think that there's a lot of value-add in making the screenshots available.
 
3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

We should document this properly, then. Not remove the feature :)
 
4) How is it supposed to be used? Do all test frameworks (JUnit/NUnit/py.test/rspec/mocha/...) allow to examine the exception occured during test execution in a kind of @After-block? No. Should a user catch an exception in every test and examine and save the screenshot if present?

It's really up to them. Personally, I use junit and a runner to capture this stuff.
 
5) And it is present there *only* if it is a remote session. If you run test locally you have to implement taking screenshots on failure anyway. So they do it in local/remove agnostic manner.

The main reason we did things this way was because people running tests locally tend to be sitting in front of the machine and can see what's happening. The remote sessions (certainly at the time, and it's still the case) are being run on machines that folks can't eyeball easily. It might be nice to add a capability to allow local implementations to take screenshots on exceptions locally.
 
My suggestion:
a) disable this feature by default, because it is not very popular, but gives performance penalty to all users,

I disagree, but perhaps we can add a flag to the server to make it easier to disable. 
 
b) may be implement it for local runs too.

I like this.

Simon
 
Alexei Barantsev,
-- 
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 0:31:15 UTC+3 пользователь Simon Stewart написал:
Same reason we include all sorts of information in our exceptions: because it's proved the best way to provide debugging information that people weren't otherwise providing. 

Normally, an exception marks the end of a test, so your example is a little contrived. The only case where I've seen people doing something that might do otherwise on a regular basis is checking to see if an element is present (by using "findElement" rather than "findElements").

In real life usage, is this causing you problems?

Simon

On Fri, Dec 23, 2016 at 2:26 PM, Alexei Barantsev <[hidden email]> wrote:
Hi, devs,

Why Selenium Server takes screenshots on exceptions by default? It's huge performance penalty.

An example:

      for (int i = 1; i < 100; i++) {
        try {
          driver.get("test.com");
        } catch (WebDriverException ex) {
        }
      }

This piece of code takes ~25 seconds and generates ~20MB of screenshots in ChromeDriver on my machine with 1920x1200 sceen resolution using the server running on the same machine.

With capability "webdriver.remote.quietExceptions" set to true the same code takes ~1,7 seconds.

15x time perfomance boost and -20MB of useless traffic.

And again the question is: why this feature is ON by default?

Regards,
-- 
Alexei Barantsev
Software-Tesing.Ru
Selenium2.Ru

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsubscrib[hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/35b7ec88-c56e-4fc0-9c48-e9a96a939959%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYHCsE4DPf_7L1REVAK5hToOJ2t06cPV%3DfMqiY3gsoBB2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Jason Leyba
On Thu, Jan 5, 2017 at 7:46 AM Simon Stewart <[hidden email]> wrote:
After a heck a pause, replying. Inline.

On Sun, Dec 25, 2016 at 9:08 AM, Alexei Barantsev <[hidden email]> wrote:
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

Let's fix it, then. Given the discussions around IDE, we can just leave that.
 
2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

That's why we write javadocs. And why things appear in stacktraces. We try and provide as much hand-holding as makes sense, but I think that there's a lot of value-add in making the screenshots available.
 
3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

We should document this properly, then. Not remove the feature :)

The feature shouldn't be removed, but it is a performance killer and should be disabled by default. We disabled it here back in 2012, and a quick search doesn't turn up anyone that has turned it back on.
 
 
4) How is it supposed to be used? Do all test frameworks (JUnit/NUnit/py.test/rspec/mocha/...) allow to examine the exception occured during test execution in a kind of @After-block? No. Should a user catch an exception in every test and examine and save the screenshot if present?

It's really up to them. Personally, I use junit and a runner to capture this stuff.
 
5) And it is present there *only* if it is a remote session. If you run test locally you have to implement taking screenshots on failure anyway. So they do it in local/remove agnostic manner.

The main reason we did things this way was because people running tests locally tend to be sitting in front of the machine and can see what's happening. The remote sessions (certainly at the time, and it's still the case) are being run on machines that folks can't eyeball easily. It might be nice to add a capability to allow local implementations to take screenshots on exceptions locally.
 
My suggestion:
a) disable this feature by default, because it is not very popular, but gives performance penalty to all users,

I disagree, but perhaps we can add a flag to the server to make it easier to disable. 
 
b) may be implement it for local runs too.

I like this.

Simon
 
Alexei Barantsev,
-- 
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 0:31:15 UTC+3 пользователь Simon Stewart написал:
Same reason we include all sorts of information in our exceptions: because it's proved the best way to provide debugging information that people weren't otherwise providing. 

Normally, an exception marks the end of a test, so your example is a little contrived. The only case where I've seen people doing something that might do otherwise on a regular basis is checking to see if an element is present (by using "findElement" rather than "findElements").

In real life usage, is this causing you problems?

Simon

On Fri, Dec 23, 2016 at 2:26 PM, Alexei Barantsev <[hidden email]> wrote:
Hi, devs,

Why Selenium Server takes screenshots on exceptions by default? It's huge performance penalty.

An example:

      for (int i = 1; i < 100; i++) {
        try {
          driver.get("test.com");
        } catch (WebDriverException ex) {
        }
      }

This piece of code takes ~25 seconds and generates ~20MB of screenshots in ChromeDriver on my machine with 1920x1200 sceen resolution using the server running on the same machine.

With capability "webdriver.remote.quietExceptions" set to true the same code takes ~1,7 seconds.

15x time perfomance boost and -20MB of useless traffic.

And again the question is: why this feature is ON by default?

Regards,
-- 
Alexei Barantsev
Software-Tesing.Ru
Selenium2.Ru

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/35b7ec88-c56e-4fc0-9c48-e9a96a939959%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYHCsE4DPf_7L1REVAK5hToOJ2t06cPV%3DfMqiY3gsoBB2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAChzDzmjL4NaMkKLYCf-Js6F7cPR%2Bei8Fxsj3gwaT03hd2ti7A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Why Selenium Server takes screenshots on exceptions by default?

Simon Stewart
On Thu, Jan 12, 2017 at 12:44 AM, Jason Leyba <[hidden email]> wrote:
On Thu, Jan 5, 2017 at 7:46 AM Simon Stewart <[hidden email]> wrote:
After a heck a pause, replying. Inline.

On Sun, Dec 25, 2016 at 9:08 AM, Alexei Barantsev <[hidden email]> wrote:
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

Let's fix it, then. Given the discussions around IDE, we can just leave that.
 
2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

That's why we write javadocs. And why things appear in stacktraces. We try and provide as much hand-holding as makes sense, but I think that there's a lot of value-add in making the screenshots available.
 
3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

We should document this properly, then. Not remove the feature :)

The feature shouldn't be removed, but it is a performance killer and should be disabled by default. We disabled it here back in 2012, and a quick search doesn't turn up anyone that has turned it back on.

I love a bit of data. Anyone want to make a command-line flag that enables/disables this feature globally, and then make the changes to capabilities processing to make sure we don't pass this anything with a DriverService if we enable per-session with a capability?

Simon
 
 
 
4) How is it supposed to be used? Do all test frameworks (JUnit/NUnit/py.test/rspec/mocha/...) allow to examine the exception occured during test execution in a kind of @After-block? No. Should a user catch an exception in every test and examine and save the screenshot if present?

It's really up to them. Personally, I use junit and a runner to capture this stuff.
 
5) And it is present there *only* if it is a remote session. If you run test locally you have to implement taking screenshots on failure anyway. So they do it in local/remove agnostic manner.

The main reason we did things this way was because people running tests locally tend to be sitting in front of the machine and can see what's happening. The remote sessions (certainly at the time, and it's still the case) are being run on machines that folks can't eyeball easily. It might be nice to add a capability to allow local implementations to take screenshots on exceptions locally.
 
My suggestion:
a) disable this feature by default, because it is not very popular, but gives performance penalty to all users,

I disagree, but perhaps we can add a flag to the server to make it easier to disable. 
 
b) may be implement it for local runs too.

I like this.

Simon
 
Alexei Barantsev,
-- 
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 0:31:15 UTC+3 пользователь Simon Stewart написал:
Same reason we include all sorts of information in our exceptions: because it's proved the best way to provide debugging information that people weren't otherwise providing. 

Normally, an exception marks the end of a test, so your example is a little contrived. The only case where I've seen people doing something that might do otherwise on a regular basis is checking to see if an element is present (by using "findElement" rather than "findElements").

In real life usage, is this causing you problems?

Simon

On Fri, Dec 23, 2016 at 2:26 PM, Alexei Barantsev <[hidden email]> wrote:
Hi, devs,

Why Selenium Server takes screenshots on exceptions by default? It's huge performance penalty.

An example:

      for (int i = 1; i < 100; i++) {
        try {
          driver.get("test.com");
        } catch (WebDriverException ex) {
        }
      }

This piece of code takes ~25 seconds and generates ~20MB of screenshots in ChromeDriver on my machine with 1920x1200 sceen resolution using the server running on the same machine.

With capability "webdriver.remote.quietExceptions" set to true the same code takes ~1,7 seconds.

15x time perfomance boost and -20MB of useless traffic.

And again the question is: why this feature is ON by default?

Regards,
-- 
Alexei Barantsev
Software-Tesing.Ru
Selenium2.Ru

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+[hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/35b7ec88-c56e-4fc0-9c48-e9a96a939959%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYHCsE4DPf_7L1REVAK5hToOJ2t06cPV%3DfMqiY3gsoBB2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAChzDzmjL4NaMkKLYCf-Js6F7cPR%2Bei8Fxsj3gwaT03hd2ti7A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYE3bWD4DHsNcE_b%2BTQfMSraXkf4fa0uwC4hU86aFZ1B-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.