.NET WebClient 403 Forbidden Error

Wasted 2 hours trying to track down the following error when making a simple WebClient DownloadFile request to an Amazon S3 url:
The remote server returned an error: (403) Forbidden.

Offending code and Url:

Uri uu = new Uri("https://zencoder-live.s3.amazonaws.com:443/add9d5d739193c13fcde60d3d7ff5ba7%2Ffe33f4d52e1cff0ef06592ed4041a7dc.mp4?Signature=b%2FXw9ylREb4up4QDw6Tyv9GyQhU%3D&Expires=1291150754&AWSAccessKeyId=AKIAIIEXNN2J4YDTRUVQ");
using (WebClient wClient = new WebClient())
{
    wClient.DownloadFile(uu, @"C:\output.mp4");
}

I was able to use Fiddler to compare Firefox's request versus my .NET application's request.

Firefox:GET /add9d5d739193c13fcde60d3d7ff5ba7%2Ffe33f4d52e1cff0ef06592ed4041a7dc.mp4?Signature=b%2FXw9ylREb4up4QDw6Tyv9GyQhU%3D&Expires=1291150754&AWSAccessKeyId=AKIAIIEXNN2J4YDTRUVQ HTTP/1.1


.NET WebClient:GET /add9d5d739193c13fcde60d3d7ff5ba7/fe33f4d52e1cff0ef06592ed4041a7dc.mp4?Signature%3Db%2FXw9ylREb4up4QDw6Tyv9GyQhU%3D%26Expires%3D1291150754%26AWSAccessKeyId%3DAKIAIIEXNN2J4YDTRUVQ HTTP/1.1

Notice that .NET is escaping my Url. Particularly the forward slash (%2F). You used to be able to pass a dontEscape parameter to the new Uri constructor but now that parameter is deprecated and is always false.

Luckily I came across a workaround on StackOverflow by Rasmus Faber:

Uri uu = new Uri("https://zencoder-live.s3.amazonaws.com:443/add9d5d739193c13fcde60d3d7ff5ba7%2Ffe33f4d52e1cff0ef06592ed4041a7dc.mp4?Signature=b%2FXw9ylREb4up4QDw6Tyv9GyQhU%3D&Expires=1291150754&AWSAccessKeyId=AKIAIIEXNN2J4YDTRUVQ");
ForceCanonicalPathAndQuery(uu);
using (WebClient wClient = new WebClient())
{
    wClient.DownloadFile(uu, @"C:\output.mp4");
}

void ForceCanonicalPathAndQuery(Uri uri){
  string paq = uri.PathAndQuery; // need to access PathAndQuery
  FieldInfo flagsFieldInfo = typeof(Uri).GetField("m_Flags", BindingFlags.Instance | BindingFlags.NonPublic);
  ulong flags = (ulong) flagsFieldInfo.GetValue(uri);
  flags &= ~((ulong) 0x30); // Flags.PathNotCanonical|Flags.QueryNotCanonical
  flagsFieldInfo.SetValue(uri, flags);
}

Now the code downloads the file like without the error!

.NET Zencoder API Wrapper

I've been working with automated video encoding systems for over 10 years. I'd like to say there have been lots of improvements in the tools available but the only tools that have made life easier are the hosted solutions like Zencoder.

Zencoder has a really nice API builder on their website and tools like John Sheehan's RestSharp make calling web services from .NET easier then ever. But why not take it a step further and get it down to one line of code to submit an encoding job? That's what I've done over my Thanksgiving weekend and the results are on GitHub in my ZencoderWrapper repository.

One liner transcode:

JobResponse job = new ZencoderClient(API_KEY).SubmitJob("http://cdeutsch.com/input.avi", "ftp://ftpuser:[email protected]/videos/", "output.mp4");

You'll need to create an account on Zencoder to use the service. And if you want to encode more then 5 seconds of video you'll need to pick a payment option. You can alternatively use the ZencoderWrapper to create your account:

CreateAccountRequest account = new CreateAccountRequest("[email protected]", "password123");

Here's a more advanced example of creating an Ogg Vorbis file with thumbnails and notifications:

ZencoderClient client = new ZencoderClient(API_KEY);
JobRequest jobRequest = new JobRequest("http://cdeutsch.com/input.avi", new OutputSetting("ftp://ftpuser:[email protected]/videos/", "output.ogg"));
//configure output settings.
jobRequest.outputs[0].audio_codec = AudioCodecSetting.Vorbis;
jobRequest.outputs[0].video_codec = VideoCodecSetting.Theora;
//add a notification.
jobRequest.outputs[0].NotificationSettings = new List<NotificationSetting>();
jobRequest.outputs[0].NotificationSettings.Add(new NotificationSetting(NotificationType.Email, "[email protected]"));
//create thumbnails
jobRequest.outputs[0].thumbnails = new ThumbnailSetting();
jobRequest.outputs[0].thumbnails.base_url = "ftp://ftpuser:[email protected]/thumbs/";
jobRequest.outputs[0].thumbnails.format = ThumbnailFormatSetting.PNG;
jobRequest.outputs[0].thumbnails.interval = 5;
jobRequest.outputs[0].thumbnails.number = 3;
jobRequest.outputs[0].thumbnails.prefix = "thumb_";
jobRequest.outputs[0].thumbnails.size = "120x80";
jobRequest.outputs[0].thumbnails.start_at_first_frame = true;
//submit the job
JobResponse job = client.SubmitJob(jobRequest);

Here's how to check the status of the job we just created:

//get job details.
JobListingResponse job = client.GetJob(job.id);

Here's how to check to the status of the job's output file (there can be multiple outputs):

//get progress of first (in this case only) output file.
JobOutputProgressResponse progress = client.GetJobOutputProgress(job.outputs[0].id);

And finally here's how to get a list of all of your jobs:

//get list of jobs
List<JobListingResponse> jobList = client.ListJobs();

If I can convince Implex (the company I work for) to outsource their encoding to Zencoder I'll be using this wrapper for their QwikCast product. I also have another personal project I plan to use the wrapper in.

If you end up using it shoot me a message I'd love the feedback!

UPDATE 11/28/2010:
Not sure how I overlooked this earlier (either missed the Zencoder "Libraries" link or just got carried away with the idea of trying something new I guess) but Chad Burggraf has already created an excellent C# Zencoder client. I may end up using his library over my own for the asynchronous support. Part of it will depend on if Chad's library can compile with Mono since the personal project I want to build will ultimately be running on OSX. I'm not sure if my library will compile in Mono "as-is", but I believe RestSharp will and I don't think I used anything special beyond what's contained in RestSharp.

WatchedIt - An MVC3, Razor, EF4 Code First Production

One of the features I love about applications like Boxee is it treats the content you're watching like email. Instead of "read" and "unread" it's "watched" and "unwatched". This is great if you don't have a significant other who tends to watch all of your shows before you. ;)

So I decided to make an application I call WatchedIt to track where I left off watching shows. What a perfect opportunity to try out some new technologies! For this project I used the following fairly new technologies:

• MVC 3 Beta
• Entity Frame 4 Code First
• Razor
• HTML 5

So what did I think of the experience? MVC3 is a promising improvement over MVC2 and Web Forms. Outside of a radical paradigm change (like being more dynamic like Ruby) I'd pick MVC3 for my next project without hesitating. It integrated nicely with jQuery by offering ways to produce and consume JSON.

HTML5 gets a big thumbs up! I've already begun using as many features as I can that are backward compatible with older browsers. If you want to know what you can and can't use now, I recommend HTML5 and CSS3 by Brian Hogan.

I really liked Razor! But the lack of intellisense (auto completion, etc) in Visual Studio was a bit painful. Hopefully this will get added soon and it will be my preferred View Engine hands down. It was a little hard to figure out when you don't have to use the @ symbol to prefix code but the error messages were very helpful.
UPDATE 11/9/2010: And 3 hours after posting this, Intellisense has been added in MVC 3 RC!

EF4 Code First didn't go nearly as smooth as the other new features even though I love the concept. The parts that worked are great but it still has a few too many limitations for me to pick it again without hesitation over other options. Some of the limitations:

• I couldn't find a simple, elegant way to do a cascading delete of all child objects when you delete a parent.
• You can't have more then one DbContext share the same database (not a major issue)
• Deploying to production didn't go very smoothly. I had to configure the connection string to use the "sa" account to get the DB created.

Keep in mind these .NET offerings are all fairly new and I think Microsoft is on the right track with all of them.

I've decided to give back to the community and put the source code on GitHub. I'm sure there are some things I could have done better or that didn't follow best practice so if you find something that would have been much easier to do another way let me know. I'd love to hear it.

Some 3rd party libraries that made this application possible:
Elmah - my favorite error logging library
tvdblib - they did an awesome job of wrapping the TVDB API!
TheTVDB - without this DB this application wouldn't be possible. Please consider contributing info and artwork.