Retrieving content: problem with non-utf-8 encoding.

Mar 17, 2009 at 5:35 PM
When i add a new story from ukrainian websites with non-unicode encoding (for example http://www.i.ua), i get such picture.

http://img150.imageshack.us/img150/7443/kiggencodingproblem.jpg

How can I solve this problem?
Coordinator
Apr 15, 2009 at 2:50 PM
Well we have plan to integrate the localization in v3.0. In the mean time you can try to implement it and let us know the issues, we will try our best to help you.
Apr 16, 2009 at 8:15 AM
 When the planned output v3.0?
Apr 17, 2009 at 3:23 AM
Edited Apr 17, 2009 at 3:24 AM

I also faced this problem when creating a site http://tumnews.ru To resolve this problem, I had a bit to change the method ReadResponse (WebRequest request) class HttpForm.  Also add a method EncodeResponse.

 private static void EncodeResponse(Uri u, string cod ,HttpFormResponse fr)
{
WebRequest wr = WebRequest.Create(u);
using (HttpWebResponse response = (HttpWebResponse)wr.GetResponse())
{
PopulateHeadersAndCookies(response, fr);
using (StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(cod)))
{
fr.Response = sr.ReadToEnd();
}
}
}

internal static HttpFormResponse ReadResponse(WebRequest request)
{
const int MaxTry = 3;

int tryCount = 0;
HttpFormResponse httpFormResponse = new HttpFormResponse();


// Sometimes the external site can throw exception so we might
// have to retry few more times
while (string.IsNullOrEmpty(httpFormResponse.Response) && (tryCount < MaxTry))
{
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
PopulateHeadersAndCookies(response, httpFormResponse);
using (StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(1251)))
{
httpFormResponse.Response = sr.ReadToEnd();
}  
}
if (httpFormResponse.Response.ToLower().IndexOf("utf-8") > -1)
{
EncodeResponse(request.RequestUri, "utf-8", httpFormResponse);
}
if (httpFormResponse.Response.ToLower().IndexOf("koi8-r") > -1)
{
EncodeResponse(request.RequestUri, "koi8-r", httpFormResponse);
}
}
catch (WebException)
{
tryCount += 1;
Thread.Sleep(200);
}
}

return httpFormResponse;
}