[DRAFT] Troubleshooting and Reporting Bugs in PhantomJS


#1

[Author’s note: I wrote this a few months ago intending to put it on the website somewhere, but never got around to doing that. Let’s polish it up a bit here and then make it a sticky thread and/or actually put it on the website. Please feel free to edit directly.]

If PhantomJS is not working correctly for you, we want to help. However, we are a small development team, and nobody is working on it full-time. We can only help you if you help us do that. This document will guide you through the process.

If you can’t figure out how to make something work

You might have found a bug in PhantomJS, but you equally might just not have gotten your own code right. Please ask for help on this forum, or the mailing list. Please do not file a bug report until you are sure you have found a bug. The bug tracker is not a support forum.

If you have the same problem as an already-reported bug

You are welcome to subscribe to that bug, but please don’t nag us by posting “+1” or “When will this be fixed?” or similar.

Many bug reports in the tracker are very old and/or do not contain a test case. If you have the same problem, it is helpful for you to report whether the problem still exists in PhantomJS 2.0, and to provide a test case, as you would for a new bug.

If you are trying to compile PhantomJS yourself

The most common reasons for people to have trouble compiling PhantomJS are inadequate RAM and missing libraries. A complete list of the software required for compilation can always be found at http://phantomjs.org/build.html.

A build using GCC may require up to 4GB of RAM per CPU core. We do not have exact numbers for Clang or Visual Studio, but we expect they are similar. If you are using GCC, the build fails, and any of these error messages appear anywhere in the build log:

g++: internal compiler error: Killed (program cc1plus)
{standard input}:nnn: Warning: end of file not at end of a line ...
{standard input}:nnn: Error: no such instruction: ...

then the problem is inadequate RAM. (There may be hundreds or even thousands of lines of spew after this error message; you will need to search the entire build log.)

You can reduce the RAM requirement to 4GB total by forcing the build to use only one CPU core:

./build.py --confirm --jobs 1 >& phantomjs-build.log

Building PhantomJS takes a very long time. Allow at least 30 minutes on a current-generation machine with 16GB of RAM and four CPU cores. Allow at least two hours on a machine with 4GB of RAM and one (usable) CPU core. Older computers will take even longer.

Can you reproduce your problem with PhantomJS 2.1?

If you are still using PhantomJS 1.9 or 2.0, the very first thing you should try is upgrading to 2.0. 2.1 includes a newer version of WebKit which fixes many bugs.

We understand that for various reasons some people cannot upgrade yet. Most of these reasons are out of our control; for instance, the NPM package of PhantomJS is not maintained by us, and we don’t know why it is still on 1.9. Issues that are under our control, like the lack of an official static binary for Linux, are being actively worked on, and we’re sorry it’s taking forever. Nonetheless, 1.9 and 2.0 are no longer being developed. Please do not report bugs unless you have reproduced them with 2.1.

Construct a minimal, complete, self-contained test case

No matter what the bug is, in order to do anything about it, we will need to be able to observe the bug for ourselves. Bug reports that don’t come with a test case are bugs we can do nothing about.

You are probably using PhantomJS to load some sort of web page and then poke at it somehow. The web page is probably large and complicated, and so is your test script. You may not be allowed to show us the whole thing. Therefore, once you have confirmed the problem exists in PhantomJS 2.1, your next step is to construct the simplest possible website and test script that still exhibit the bug.

It’s usually easiest to do this starting with the test script. Find the shortest possible sequence of PhantomJS operations that makes the bug happen. Very often all that is needed is

var p = require("webpage").create();
p.open("http://your.site.example/path", function () {
    phantom.exit(0); // we never get here, because PhantomJS crashes
});

or

var p = require("webpage").create();
p.open("http://your.site.example/path", function () {
    p.render("output.png");
    phantom.exit(0);
});
// the problem will be visible in output.png

If you are using WebDriver/Selenium, or any other kind of generic scripted-browsing library, try to replace that with a native PhantomJS script. This should always be possible, unless the bug is in our WebDriver support.

Now, go after server-side processing. Can you make a static website that exhibits the bug? If you can’t—if you need, for instance, special HTTP response headers, or a form submission and response—write a CGI script (remember CGI scripts?) that does the absolute minimum necessary, and make everything else static.

It is okay for this test website to refer to well-known public resources such as the jQuery CDN or Google Fonts, but it is better if it includes static copies of everything needed. This is because public resources could change or go away at any time and that might make the bug unreproducible. Don’t include anything you don’t have permission to give us a copy of, and don’t include anything that we couldn’t stick in our automated test suite (and therefore make available to the whole world in our source distribution). Your corporate logo is probably not essential to making the bug happen. (If it is, there’s probably something weird with the image file, and you should be able to synthesize an image file with the same weirdness but without any corporate IP.)

Now start cutting things out of the static website. Remove every HTML element that isn’t involved in the problem. Remove as much text as possible. Replace all the remaining text with lorem ipsum, or quotes from Moby-Dick, or something else that’s in the public domain. Bugs almost never depend on the exact sequence of characters (they can, however, depend on the writing system; if your website is in Kannada, you may need to use Kannada lorem ipsum). Replace all the images with placeholders. Do the same thing for the page’s CSS and JavaScript. You are shooting for less than a kilobyte of HTML, with no external references. This won’t always be possible, but it usually is.

Here’s an example of a well-minimized test HTML file:

<!doctype html>
<!-- Crash on load -->
<script>
window.onload = function boom()
{
  document.getElementsByTagName("tbody")[0].style.position = "absolute";
  document.getElementsByTagName("table")[0].style.color = "green";
}
</script>
<body>
<table><tbody></tbody></table>

If you’re using PhantomJS for something other than loading a web page and poking at it, apply the same logic to whatever it is you are doing.

After you have a minimal test script and a minimal HTML page (plus any external resources, CGI scripts, etc. that cannot be avoided), the last step is to write an explanation of the test. We need to know, in detail, what happens for you and what you think should happen instead. It is not enough to say “It doesn’t work,” because we don’t know what “working” means for you. We need something like “This HTML crashes PhantomJS,” or “The red box on this page is supposed to be green, and it is green in Firefox and IE >=9,” or “This form submission comes through to the server in ISO-8859-1 even though I thought I told it to use UTF-8.”

Is the bug in PhantomJS proper?

Historically, the vast majority of bugs reported to the PhantomJS issue tracker have been bugs in WebKit. We do not have either the manpower or the expertise required to fix these bugs ourselves; we do, however, regularly upgrade our copy of WebKit.

Bugs which are likely to be in WebKit include:

  • Crashes, especially crashes where the first few lines of the stack trace mention “JSC”, “WebCore”, “WebKit”, “WTF”, or “KSomething”.
  • Any situation where PhantomJS does not render a webpage the same way other browsers do.
  • Missing Web platform features, e.g. JavaScript/DOM APIs, CSS constructs, HTML elements and attributes, file formats (image, font, video, etc).

If you have a Mac, you can test whether your minimized test case is handled correctly by a nightly build of WebKit. Many Linux distributions include a minimalist web browser based on Webkit (often called something like “Epiphany”, “Kmeleon”, or “Web”) that can also be used for this. Unfortunately, testing with Safari or Chrome is usually not reliable.

If you determine that your bug is in WebKit, please report it directly to the Webkit developers and not to us.

The bug could also be in Qt. Bugs which are likely to be in Qt include:

  • Crashes where the first few lines of the stack trace mention “QSomething”
  • Bugs having something to do with the network traffic produced by PhantomJS
  • Bugs in the implementation of HTTP
  • Missing network stack features (TLS, HTTP/2.0, etc)

We have more ability to do something ourselves about bugs in Qt, so you should report those bugs to us, but if you’re sure it’s a bug in Qt, it can be worthwhile to report it directly to the Qt project as well.

Finally, Qt and Webkit depend on many other open-source libraries, and your bug could be in those as well. As with Webkit, if you know that your bug is in one of those libraries, it’s most efficient to bypass us and report it directly to the developers of the offending library. However, figuring out which library is at fault can be very difficult, so don’t worry about it if you aren’t sure.

Stack traces and crash dumps are usually useless

Older versions of PhantomJS (prior to 2.1) would generate a “crash dump” file when they crashed, and would encourage you to send that dump to us, or at least extract a stack trace from it and send that in. Please don’t do that.

A stack trace (or a crash dump), all by itself, almost never has enough information to let us fix the bug. We will still have to ask you for a test case. And if we have a test case, we don’t need the stack trace, because we can watch the crash happen ourselves. So, please just send in a test case to begin with.

Stack traces can help you figure out whether a bug should be reported to WebKit or Qt instead of us, but it’s usually easy enough to figure that out from a test case, and the Webkit and Qt people also want a test case. So, again, start with the test case.

Reporting bugs at Github

[to be written]


#2

Issue about issue templates on Github: https://github.com/ariya/phantomjs/issues/14016

Let’s construct the template?

We surely need these info:

  1. PhantomJS version;
  2. Operating system;
  3. Write a minimal test case to reproduce your problem.

#3