ASP.NET 中抓取网页内容 并出现服务器提交了协议冲突. Section=ResponseHeader 解决方案

ASP.NET 中抓取网页内容是非常方便的，而其中更是解决了 ASP 中困扰我们的编码问题。

需要三个类：WebRequest、WebResponse、StreamReader。

WebRequest、WebResponse 的名称空间 是：

System.Net

StreamReader 的名称空间是：

System.IO

核心代码

WebRequest request = WebRequest.Create("http://www.cftea.com/");

WebResponse response = request.GetResponse();

StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding("gb2312"));

• WebRequest 类的 Create 为静态方法，参数为要抓取的网页的网址；
• Encoding 指定编码，Encoding 中有属性 ASCII、UTF32、UTF8 等全球通用的编码，但没有 gb2312 这个编码属性，所以我们使用 GetEncoding 获得 gb2312 编码。

示例

    private static string getContent(string Url)

        {

            string strResult = "";

            try

            {

                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(Url);

                //声明一个HttpWebRequest请求  

                request.Timeout = 30000;

                //设置连接超时时间  

                request.Headers.Set("Pragma", "no-cache");

                HttpWebResponse response = (HttpWebResponse)request.GetResponse();

                Stream streamReceive = response.GetResponseStream();

                Encoding encoding = Encoding.GetEncoding("GB2312");

                StreamReader streamReader = new StreamReader(streamReceive, encoding);

                strResult = streamReader.ReadToEnd();

                streamReader.Close();

            }

            catch

            {

                throw;

            }

            return strResult;

        }

        private string GetUrl(string url)

        {

            string str = string.Empty;

            try

            {

                WebRequest request = WebRequest.Create(url);

                WebResponse response = request.GetResponse();

                StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding("gb2312"));

                str = reader.ReadToEnd();

                reader.Close();

                reader.Dispose();

                response.Close();

                return str;

            }

            catch (Exception ex)

            {

                str = ex.Message;

                return str;

            }

        }

        private string GetPostContent(string strUrl)

        {

            string strMsg = string.Empty;

            try

            {

                string data = "";

                byte[] requestBuffer = System.Text.Encoding.GetEncoding("gb2312").GetBytes(data);

                WebRequest request = WebRequest.Create(strUrl);

                request.Method = "POST";

                request.ContentType = "application/x-www-form-urlencoded";

                request.ContentLength = requestBuffer.Length;

                using (Stream requestStream = request.GetRequestStream())

                {

                    requestStream.Write(requestBuffer, 0, requestBuffer.Length);

                    requestStream.Close();

                }

                WebResponse response = request.GetResponse();

                using (StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding("gb2312")))

                {

                    strMsg = reader.ReadToEnd();

                    reader.Close();

                }

            }

            catch

            { }

            return strMsg;

        }

一般情况下会出现这个问题  解决如下

服务器提交了协议冲突.   Section=ResponseHeader   Detail=CR   后面必须是   LF  

The   server   committed   a   protocol   violation.   Section=ResponseHeader   Detail=CR   must   be   followed   by   LF

主体意思是微软没有容忍不符合RFC   822中的httpHeader必须以CRLF结束的规定的服务器响应。

一个解决方案是在application.config或web.config文件里加入

    <system.net>

        <settings>

            <httpWebRequest   useUnsafeHeaderParsing= "true "   />

        </settings>

    </system.net>

允许系统容忍（tolerant）只以CR或LF结尾的hearder信息