利用webservice和.net技术上传和下载文件--Sending files in chunks with MTOM Web Services and .NET 2.0 from:http://www.codeproject.com/soap/MTO

98 votes for this article. Popularity: 9.58. Rating: 4.81 out of 5.

Download source for web service, windows client and class library - 203.6 Kb Introduction In trying to keep up to speed with .NET 2.0, I decided to do a .NET 2.0 version of my CodeProject article "DIME Buffered Upload", which used the DIME standard to transfer binary data over web services. The DIME approach was reasonably efficient, but the code is quite complex and I was keen to explore what .NET 2.0 has to offer. In this article, I use version 3.0 of the WSE (Web Service Enhancements), which is available for .NET 2.0 as an add-in, to provide a simpler and faster method of sending binary data in small chunks over HTTP web services. Background Just a re-cap on why you may need to send data in small chunks at all: if you have a large file and you want to send it across a web service, you must understand the way it all fits together between IIS, .NET and the web service call. You send your file as an array of bytes as a parameter to a web service call, which is all sent to the IIS web server as a single request. This is bad if the size of the file is beyond the configured MaxRequestLength of your application or if the request causes an IIS timeout. It is also bad from the point of view of providing feedback of the file transfer to the user interface because you have no indication of how the transfer is going until it is either completed or failed. The solution outlined here is to send chunks of the file one by one and append them to the file on the server. There is an MD5 file hash done on the client and the server to verify that the file received is identical to the file sent. Also, both upload and download file transfers are included in this article. Adventures with MTOM MTOM stands for SOAP "Message Transmission Optimization Mechanism" and it is a W3C standard. To use it and to run this application, you must download and install WSE 3.0, which includes MTOM support for the first time. If you look in the app.config and web.config files in the source code, you will see sections referring to the WSE 3 assembly and a messaging clientMode or serverMode setting. These are necessary to run MTOM in the application. The problem with DIME was that the binary content of the message was sent outside the SoapEnvelope of the XML message. This meant that although your message was secure, the Dime Attachment may not be secure. MTOM fully complies with the other WS-* specifications like WS-Security, so the entire message is secure. It took me a while to realise that when MTOM is turned on for the client and the server, WSE automatically handles the binary encoding of the data in the web service message. With DIME and WSE 2.0, you had to code your application for DIME by using DimeAttachments. This is no longer necessary; you just send your byte[] as a parameter or a return value to WebMethod and WSE makes sure that it is sent as binary, not padded by XML serialization as it would be in the absence of DIME or MTOM. The user interface The client application should be fairly straightforward. There are options at the top of the form to indicate if you want the file hash check done at the end of the transfer. You can also manually set the chunk size or you can tick the box to AutoSet to let it regulate itself. In general, I like using status bars more than message-boxes, as they are less intrusive. So keep an eye on the status bar to see what's going on. Then there is the panel for uploading a file. Simply click browse, click the button and then you're away. The panel for downloading files is similarly easy to use. It has a listbox showing all of the files in the Upload folder on the server; there probably won't be any there by default. Just select a file and click download. You can refresh the files list at any time with the refresh button. To change the save folder, enter a new path in the text box provided. Lastly, there is a textbox at the bottom which will contain the file hashes of the local and remote files after the transfer is complete. Use the File menu for some additional options, like manually resuming a transfer and manually invoking a file hash check. A forms-authenticated web service? You can also tick the box for "Login Required" if your website is configured with forms authentication. Normally, you can't use a web service if it is protected by forms authentication. This is because forms authentication is performed via a login ASPX page and an authentication cookie is given to the client browser. These conditions are not webservice friendly. There is a work-around to allow you to protect the webservice via forms authentication. It sends HttpWebRequest to the login.aspx page and captures the cookie, placing it in the web service objects. Look in the code-behind of login.aspx.cs to see the minor modification that was needed to accept a login via a query string, i.e. HttpWebRequest. To use forms authentication, just change the authentication section of web.config, which is set to Windows authentication by default. Then you will need to enter a username and password -- admin/admin by default -- in the client application to upload or download a file. How the code works The web service has two main methods: AppendChunk is for uploading a file to the server and DownloadChunk is for downloading from the server. These methods receive parameters for the file name, the offset of the chunk, and the size of the buffer being sent/received. The Windows Forms client application can upload a file by sending all of the chunks one after the other using AppendChunk until the file has been completely sent. It does an MD5 hash on the local file and compares it with the hash of the file on the server to ensure that the contents of the files are identical. The download code is very similar, the main difference being that the client must know from the server how big the file is so that it can know when to stop requesting chunks. A simplified version of the upload code from the WinForms client is shown below. Have a look in the code for Form1.cs to see the inline comments and the explanation of the code. Essentially, a file stream is opened on the client for the duration of the transfer. Then the first chunk is read into the Buffer byte array. The while loop keeps running until the FileStream.Read() method returns 0, i.e. the end of the file has been reached. For each iteration, the buffer is sent directly to the web service as a byte[]. The 'SentBytes' variable is used to report progress to the form. using(FileStream fs = new FileStream(LocalFilePath, FileMode.Open, FileAccess.Read)) { int BytesRead = fs.Read(Buffer, 0, ChunkSize); while(BytesRead > 0 && !worker.CancellationPending) { ws.AppendChunk(FileName, Buffer, SentBytes); SentBytes += BytesRead; BytesRead = fs.Read(Buffer, 0, ChunkSize); } } Setting the chunk size In many Windows Forms applications, regular feedback to the user is very important. Having a responsive and visually communicative application is usually worth a small sacrifice in performance. Feedback for file transfers is typically done via a progress bar and/or status bar message. Obviously, the web services aspect to a chunked file transfer is overhead. The client constructs and sends the SOAP message and then the server receives and parses it before sending the response. If the chunk size is very small, i.e. 2Kb, then there is a lot of messaging going on and not much data transfer. It should be clear then that we should aim for the highest possible chunk size that is within our requirements for quick user interface feedback. I have aimed for each chunk to be completed in 800 milliseconds. You can adjust this setting programmatically before the file transfer. See the PreferredTransferDuration variable in the file transfer object. The client regulates the chunk size automatically, to ensure that each chunk is completed in the desired time. This is done by sampling every 15th chunk -- also adjustable, ref ChunkSizeSampleInterval -- and adjusting the chunk size based on the time it takes to transfer this chunk. The overall result is a self-controlled file transfer that will adapt to changing network conditions during the transfer. One useful feature is that the webservice provides the MaxRequestLength setting on the server, which the client retrieves before the transfer in order to stay within acceptable request sizes on the server. The client indicates "Max" in the status bar if the chunk size is at its maximum. Resume transfers supported The application also supports resuming a failed upload or download. I use this application to copy a system image (20+ gigs) from my web server to my home PC. Obviously, there is a good chance of a connection being dropped during such a lengthy transfer, even with a 3Mbit bidirectional line. Resume support is a must when dealing with such large files and fortunately, it is very simple to include this in the application. Because data is only written to the file after it has been successfully received, we can be confident in resuming a file transfer based on the size of the partial file. In practice, this works perfectly. Click File > Resume Upload or File > Resume Download etc. to locate the partial file you wish to resume. After the file transfer, an MD5 hash is requested from the server and the client to compare the files. The timeout value is increased for the hash check, but it is always possible that it will timeout when checking such a large file. So you might have transferred the correct number of bytes, but have no way of verifying that they are the right bytes. I have included a manual MD5 hash check, available in the File menu. If the server is not giving the file hash within the timeout limit, you could run the client application directly on the server and check it locally, thus overcoming the timeout problem. Incorporating into your own application To use this code in your own client application, simply add a reference to MTOM_Library.dll, which is included in the article download. Then you should use the example client application as a starting point. It shows how to use the classes provided in order to perform the file transfer. Refer to the SetUpDownload and SetUpUpload methods of Form1.cs in particular. You use the FileTransferUpload class to do an upload and the FileTransferDownload class to do a download. Each class has settings such as ChunkSize, AutoSetChunkSize, LocalSaveFolder, IncludeHashVerification, etc. You can configure these as needed in your code. To change the location of the web service, make sure that the app.config file is deployed to your run directory. Change the MTOM_Library_MtomWebService_MTOM setting to the location of your web service. Your ASP.NET application will also need to host the MTOM.asmx web service and the Login.aspx code behind if you use forms authentication. To configure the web application for MTOM, just copy the settings from the web.config file included in this application. You can change the Upload folder on the server in web.config, also. Files are downloaded from and uploaded to this folder. Points of interest The BackgroundWorker class in .NET 2.0 .NET 2.0 has a great new class called BackgroundWorker to simplify running tasks asynchronously. Although this application sends the file in small chunks, even these small chunks would delay the WinForms application and make it look crashed during the transfer. So, the web service calls still need to be done asynchronously. The BackgroundWorker class works using an event model, where you have code sections to run for DoWork (when you start), ProgressChanged (to update your progress bar / status bar) and Completed (or failed). You can pass parameters to the DoWork method, which you could not do with the Thread class in .NET 1.1. I know you could it with delegates, but delegates aren't great for thread control. You can also access the return value of DoWork in the Completed event handler. So for once, MS has thought of everything and made a very clean threading model. Exceptions are handled internally and you can access them in the Completed method via the RunWorkerCompletedEventArgs.Error property. The code shown below is an example of the ProgressChanged event handler: private void workerUpload_ProgressChanged(object sender, ProgressChangedEventArgs e) { // update the progress bar and status bar text this.toolStripProgressBar1.Value = e.ProgressPercentage; this.statusText.Text = e.UserState.ToString(); // summary text is sent in the UserState parameter } I have used four BackgroundWorker objects in the application: One to manage the upload process One to manage the download process One to calculate the local MD5 file hash in parallel while waiting for the server result One to download the list of files in the Upload server folder, allowing the user to select a file to download The reason I use a BackgroundWorker object for each task is because the code for each task is tied into the events for that object and I like this programming model. A good example of Thread.Join() When the upload or download is complete, the client asks for an MD5 hash of the file on the server. It can thus compare it with the local file to make sure they are identical. I originally did these in sequence, but it can take a few seconds to calculate the result for a large file, i.e. anything over a few hundred Mb. So, the application was waiting five seconds for the server to calculate the hash, and then five more seconds for the client to calculate its own hash. This made no sense, so I decided to implement a multi-threaded approach to allow them to run in parallel. While the client is waiting on the server, it should be calculating its own file hash. This is done with the Thread class and the use of the Join() method, which blocks execution until the thread is complete. The code below shows how this is accomplished: // start calculating the local hash (stored in class variable) this.hashThread = new Thread(new ThreadStart(this.CheckFileHash)); this.hashThread.Start(); // request the server hash string ServerFileHash = ws.CheckFileHash(FileName); // wait for the local hash to complete this.hashThread.Join(); if(this.LocalFileHash == ServerFileHash) e.Result = "Hashes match exactly"; else e.Result = "Hashes do not match"; There is a good chance that the two operations will finish at approximately the same time, so very little waiting around will actually happen. Common problems and questions Visual Studio compile errors There have been dozens of questions about people not being able to compile the solution in Visual Studio. You may get an error like this: The type or namespace name 'MTOMWse' does not exist in the namespace 'UploadWinClient.MtomWebService'. With an MTOM-enabled web service, Visual Studio is supposed to generate 2 proxy classes: a standard one derived from System.Web.Services.Protocols.SoapHttpClientProtocol and a WSE class derived from Microsoft.Web.Services3.WebServicesClientProtocol, with "Wse" tagged onto the end of the proxy class name. Sometimes VS misbehaves and does not generate this class. I don't understand why, but the work-around is to Show-all-files in the WinForms project and expand the web service > Reference.map > Reference.cs. Edit this file and change public partial class MTOM : System.Web.Services.Protocols.SoapHttpClientProtocol to public partial class MTOMWse : Microsoft.Web.Services3.WebServicesClientProtocol. Also make sure to update the constructor to match the new class name. Then it should compile fine. Can you make a web client? No, no and double no! I have gotten a ton of questions about people asking for a web client instead of a WinForms client. This is fundamentally impossible because of the advanced file Input/Output required to achieve this solution. For good reason, browsers do not provide this level of access to the file system of the client. A guy called Brettle wrote a progress bar control for ASP.NET file uploading. This may be your best bet, although you must understand that web applications are very limited when it comes to sending large amounts of data to the web server. Conclusions I found that MTOM was about 10% faster than DIME in my limited testing. This is probably to do with the need to package up each chunk into a DIME attachment, which is no longer necessary with MTOM. Remember, if you want to send chunks larger than 4 Mb, you must increase the .NET 2.0 Max Request Size limit in your web.config file. Feel free to use this code and modify it as you please. Please post a comment for any bugs, suggestions, or improvements. Enjoy! History 29 Dec 2005 - Original version posted 14 Feb 2007 - Updated source download 17 July 2007 - Updated source download; also added app.config entry for MTOM ClientMode=On, otherwise MTOM is not used! Tim_Mackey Click here