日期:2014-05-17 浏览次数:20813 次
问题描述,使用htmlparser的lexer解析器进行页面解析时发现类似如下的页面会有问题:
?
?
<script>
for(i=0;i<a;i++){
}
</script>
解析后代码变成了:
<script>
for(i=0;i<a;i++){
}
></script>
?
?通过lexer代码发现,实际上只要js代码改成:
<script>
<!--
for(i=0;i<a;i++){
}
-->
</script>
?就不会有问题了,从代码中主要发现它的解析其实没有问题,主要是我们平时的页面规范做的不好,它在解析时会看到字符解析时发现<后面如果有字母就认为它是一个tag:
protected Node parseString(int start, boolean quotesmart)
throws ParserException {
boolean done;
char ch;
char quote;
done = false;
quote = 0;
while (!done) {
ch = mPage.getCharacter(mCursor);
if (Page.EOF == ch)
done = true;
else if (0x1b == ch) // escape
{
ch = mPage.getCharacter(mCursor);
if (Page.EOF == ch)
done = true;
else if ('$' == ch) {
ch = mPage.getCharacter(mCursor);
if (Page.EOF == ch)
done = true;
// JIS X 0208-1978 and JIS X 0208-1983
else if ('@' == ch || 'B' == ch)
scanJIS(mCursor);
/*
* // JIS X 0212-1990 else if ('(' == ch) { ch =
* mPage.getCharacter (mCursor); if (Page.EOF == ch) done =
* true; else if ('D' == ch) scanJIS (mCursor); else {
* mPage.ungetCharacter (mCursor); mPage.ungetCharacter
* (mCursor); mPage.ungetCharacter (mCursor); } }
*/
else {
mPage.ungetCharacter(mCursor);
mPage.ungetCharacter(mCursor);
}
} else
mPage.ungetCharacter(mCursor);
} else if (quotesmart && (0 == quote)
&& (('\'' == ch) || ('"' == ch)))
quote = ch; // enter quoted state
// patch from Gernot Fricke to handle escaped closing quote
else if (quotesmart && (0 != quote) && ('\\' == ch)) {
ch = mPage.getCharacter(mCursor); // try to consume escape
if ((Page.EOF != ch) && ('\\' != ch) // escaped backslash
&& (ch != quote)) // escaped quote character
// ( reflects ["] or ['] whichever opened the quotation)
mPage.ungetCharacter(mCursor); // unconsume char if char not
// an escape
} else if (quotesmart && (ch == quote))
quote = 0; // exit quoted state
else if (quotesmart && (0 == quote) && (ch == '/')) {
// handle multiline and double slash comments (with a quote)
// in script like:
// I can't handle single quotations.
ch = mPage.getCharacter(mCursor);
if (Page.EOF == ch)
done = true;
else if ('/' == ch) {
do
ch = mPage.getCharacter(mCursor);
while ((Page.EOF != ch) && ('\n' != ch));
} else if ('*' == ch) {
do {
do
ch = mPage.getCharacter(mCursor);
while ((Page.EOF != ch) && ('*' != ch));
ch = mPage.getCharacter(mCursor);
if (ch == '*')
mPage.ungetCharacter(mCursor);
} while ((Page.EOF != ch) && ('/' != ch));
} else
mPage.ungetCharacter(mCursor);
} else if ((0 == quote) && ('<' == ch)) {
ch = mPage.getCharacter(mCursor);
if (Page.EOF == ch)
done = true;
// the order of these tests might be optimized for speed:
else if ('/' == ch
|| Character.isLetter(ch)
|| '!' == ch || '%' == ch || '?' == ch) {
done = true;
mPage.ungetCharacter(mCursor);
mPage.ungetCharacter(mCursor);
} else {
// it's not a tag, so keep going, but check for quotes
mPage.ungetCharacter(mCursor);
}
}
}
return (makeString(start, mCursor.getPosition()));
}
?因此为了解决这个问题,现在要在上面做一个手脚:
首先在类中间增加了一个标记,script
这个标记是修改了nexNode方法,在返回前判断下是否前一个标签是<script> 或者</script>
然后在parseString中修改其解析方法就可以了,下面是完整的代码:
import java.net.URLConnection;
import org.htmlparser.Node;
import org.htmlparser.lexer.Lexer;
import org.htmlparser.lexer.Page;
import or