A CreateRiaClientFilesTask gotcha

Got a Silverlight project that builds fine on my computer, however, on a co-worker's kept failing with CreateRiaClientFilesTask cannot load asssembly or what-not. The gotcha -- I had set the web.config to impersonate a system account and that system account should have at least Modify access for the 'Temporary ASP.NET Files' folder.