It's hard to say which one is better than the other. It is even more difficult because I am a Batman fan from way back and love Watchmen. Both performances were amazing, but I am not sure it is fair to either actor to compare the two. To preface what I am about to say, I will admit that I think Watchmen was a better movie than The Dark Knight. This is not because I am a bigger Watchmen fan, but because on repeat viewings The Dark Knight has plot holes that are too large to ignore and I was extremely dissatisfied with the ending; not that Batman had to run, but that it was a stupid solution to the problem of Harvey Dent/Two-Face...in Watchmen, however, the ending made sense and worked; I also feel as a whole the performances in Watchmen were stronger than the performances in The Dark Knight.
So anyway, on to my comparison:
The Dark Knight, while drawing inspirations from past Batman comics/cartoons was largely original and gave Ledger a lot of room to develop the character and do his own thing with it. He was not restricted to a very limited history or source material. Not only that, but he also had a number of past performances to draw from and expand upon, most notably Nicholson in 'Batman' and Hamil in 'The Animated Series,' but excellent performances in their own right/medium. Furthermore, Ledger had the opportunity the rely on his entire body and face in his portrayal, as well as the excellent makeup job.
Jackie Earle Haley, on the other hand, had a very limited source material and could not deviate from what was within the Graphic Novel. Nearly all of his lines, actions, etc. were restricted to panels in a Comic Book. So not only was he restricted in that sense, but he had to convey emotion and get the audience to connect with his character through a mask that remained on for most of the film.
Both actors truly inhabited and became the character they were portraying. Both did a great job and gave powerful performances, but I think overall Jackie Earle Haley as Rorschach/Walter Kovacs was the better performance. The emotion he put into the role and conveyed, both with and without the mask, was incredible. The scene where he killed Grice was such a nuanced and powerfully emotional scene that I got chills. Even with the mask, you could sense that his face was truly twisted in pain and he was conflicted with what he was doing. The shaking, the posture, the stance...and each strike was deftly executed with the utmost restraint and pain. And the scenes without his mask, most notably when he was having the mask ripped off by the police and his death, were very important scenes and I was worried they would not translate well to the screen; boy was I wrong. Haley nailed the performance, and I feel, gave the far superior performance of the two.
That being said, I do not think he will win an Oscar for this performance.