怎樣使用C獲取百度搜索結(jié)果,并且解析到目標(biāo)網(wǎng)址

1、怎樣使用C# 獲取百度搜索結(jié)果，并且解析到目標(biāo)網(wǎng)址我們首先應(yīng)該分析百度的搜索結(jié)果，發(fā)現(xiàn)百度的搜索結(jié)果的格式為：圖中標(biāo)記部分可以知道，百度的搜索結(jié)果都是在id=”content_left”的結(jié)果中的，每個(gè)搜索項(xiàng)目的是以class=”result c-container”作為一項(xiàng)， 每項(xiàng)中的題目又是包含在h3標(biāo)簽中，如下圖所示：因此我們有了思路：1. 根據(jù)關(guān)鍵字獲取到百度搜索結(jié)果的整個(gè)HTML文本2. 正則匹配到搜索結(jié)果容器的HTML3. 正則匹配到搜索結(jié)果每一項(xiàng)的HTML4. 取出每項(xiàng)結(jié)果中的題目和鏈接地址直接來干的，看下面的代碼：using System; using System.Colle

2、ctions.Generic; using System.Text; using System.Text.RegularExpressions; using System.Web; using System.Net;using System.IO;namespace BaiduSearchTest struct BaiduEntry public string title, brief, link; class Program static string GetHtml(string keyword) string url = " string encodedKeyword = Ht

3、tpUtility.UrlEncode(keyword, Encoding.GetEncoding(936); /百度使用codepage 936字符編碼來作為查詢串，果然專注于中文搜索 /更不用說，還很喜歡微軟 /谷歌能正確識(shí)別UTF-8編碼和codepage這兩種情況，不過本身網(wǎng)頁在HTTP頭里標(biāo)明是UTF-8的 /估計(jì)谷歌也不討厭微軟（以及微軟的專有規(guī)范） string query = "s?wd=" + encodedKeyword;  HttpWebRequest req; HttpWebResponse response; Stream stream;

4、 req = (HttpWebRequest)WebRequest.Create(url + query); response = (HttpWebResponse)req.GetResponse(); stream = response.GetResponseStream(); int count = 0; byte buf = new byte8192; string decodedString = null; StringBuilder sb = new StringBuilder(); try Console.WriteLine("正在讀取網(wǎng)頁0的內(nèi)容", url

5、+ query); do count = stream.Read(buf, 0, buf.Length); if (count > 0) decodedString = Encoding.GetEncoding("utf-8").GetString(buf, 0, count); sb.Append(decodedString); while (count > 0); catch Console.WriteLine("網(wǎng)絡(luò)連接失敗，請(qǐng)檢查網(wǎng)絡(luò)設(shè)置。"); return sb.ToString(); static void PrintResul

6、t(List<BaiduEntry> entries) int count = 0; entries.ForEach(delegate(BaiduEntry entry) Console.WriteLine("找到了百度的第0條搜索結(jié)果：", count += 1); if (entry.link != null) Console.WriteLine("找到了一條鏈接："); Console.WriteLine(entry.link); if (entry.title != null) Console.WriteLine("標(biāo)題為：

7、"); Console.WriteLine(entry.title); if (entry.brief != null) Console.WriteLine("下面是摘要："); Console.WriteLine(entry.brief); Program.Cut(); ); static void simpleOutput() string html = "<table><tr><td><font>test</font><a>hello</a><br>&l

8、t;/td></tr></table>" Console.WriteLine(RemoveSomeTags(html); static string RemoveVoidTag(string html) string filter = "<br>" ; foreach (string tag in filter) html = html.Replace(tag, ""); return html; static string ReleaseXmlTags(string html) string filt

9、er = "<a.*?>", "</a>", "<em>", "</em>", "<b>", "</b>", "<font.*?>", "</font>" ; foreach (string tag in filter) html = Regex.Replace(html, tag, ""); return html; &

10、#160; static string RemoveSomeTags(string html) html = RemoveVoidTag(html); html = ReleaseXmlTags(html); return html; static void Cut() Console.WriteLine(""); static void MainProc(string input) MainProc(input, false);   static void MainProc(string input, bool tagsForBrief) Regex r = n

11、ew Regex("<h3sS*?</h3>", RegexOptions.IgnoreCase);  MatchCollection matchCollection = r.Matches(input); List<string> collection = new List<string>();  foreach(Match m in matchCollection)   string textReg = "<as*>*>(sS+?)</a>"

12、0; MatchCollection textMatchCollection = Regex.Matches(m.Value, textReg, RegexOptions.IgnoreCase);  foreach (Match match in textMatchCollection) if (match.Success) Console.Write(match.Result("$1");   string LinkReg = "http:/(w-+.)+w-+(/w- ./?%&=*)?"  MatchColle

13、ction linkMatchCollection = Regex.Matches(m.Value, LinkReg, RegexOptions.IgnoreCase);  foreach (Match match in linkMatchCollection) if (match.Success) Console.Write(match.Groups0.Value); public static void Main(string args) Console.WriteLine("請(qǐng)輸入一個(gè)關(guān)鍵字。"); string keyword; keyword = Con

14、sole.ReadLine(); Console.WriteLine("正在從百度上獲取結(jié)果，請(qǐng)稍等"); string input; input = GetHtml(keyword); Regex r = new Regex("<div id="content_left"sS*</div><div style="clear:both;height:0;"></div>", RegexOptions.IgnoreCase); input = r.Match(input).V

15、alue; MainProc(input); Console.ReadKey(true); 程序結(jié)果如下圖所示：通過上面的例子你應(yīng)該明白怎樣使用.NET/C# 獲取百度搜索結(jié)果項(xiàng)了吧，程序可以直接使用，如果沒有得到結(jié)果說明是百度搜索的結(jié)構(gòu)變了，請(qǐng)按程序思路改正。Ok, 我們看到此時(shí)的搜索結(jié)果的url 都是被 被百度重定向過后的url 地址。 都是以 開頭的。那么我們需要再多做一點(diǎn)才能得到真正的url地址。我們用C# Get一下發(fā)現(xiàn)其實(shí)訪問的是一個(gè)包含我們目標(biāo)網(wǎng)址的一個(gè)網(wǎng)頁。再多用一次正則去或者真正的目標(biāo)網(wǎng)頁的url就好了。 public static string GetTheRedirect

16、Url(string originalAddress) string redirectUrl="" HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(originalAddress); HttpWebResponse response = (HttpWebResponse)request.GetResponse(); redirectUrl =(String) response.ResponseUri.AbsoluteUri; Stream ReceiveStream = response.GetR

17、esponseStream(); Encoding encode = System.Text.Encoding.GetEncoding("utf-8"); StreamReader readStream = new StreamReader(ReceiveStream, encode); Char read = new Char256; / Read 256 charcters at a time. int count = readStream.Read(read, 0, 256); Console.WriteLine("HTML.rn"); while (count > 0) String str = new String(read, 0, count); count = readStream.Read(read, 0, 256); String patternstr = "url=s*(?:"(?<1>"*)"|(?<1>S+)"></noscript>" Regex pattern = new Regex( patternstr, RegexOption