天天看点
返回

我心中的核心组件（可插拔的AOP）~分布式文件上传组件~基于FastDFS

回到目录

一些概念

在大叔框架里总觉得缺点什么，在最近的项目开发中，终于知道缺什么了，分布式文件存储组件，就是缺它，呵呵，对于分布式文件存储来说，业界比较公认的是FastDFS组件，它自己本身就是集群机制，有自己的路由选择和文件存储两个部分，我们通过FastDFS的客户端进行上传后，它会返回一个在FastDFS上存储的路径，这当然是IO路径，我们只要在服务器上开个Http服务器，就可以以Http的方法访问你的文件了。

我的组件实现方式

前端上传控件（表单方式，swf方式，js方法均可）将文件流传给我们的FastDFS客户端，通过客户端与服务端建立Socket连接，将数据包发给FastDFS服务端并等待返回，上传成功后返回路径，我们可以对路径进行ＨＴＴＰ的处理，并存入数据库

fastDFS配合nginx服务器自动生成指定尺寸的图像

原图像地址： http://www.fastdfs.com/demo/pictruename.jpg

指定尺寸的图像地址：http://www.fastdfs.com/demo/pictruename_100x100.jpg

技术实现

１　一个接口，定义三种上传规格，普通文件，图像文件和视频文件（一般需要对它进行截力）

public interface IFileUploader
    {
        /// <summary>
        /// 上传视频文件
        /// </summary>
        /// <param name="param"></param>
        /// <returns></returns>
        VideoUploadResult UploadVideo(VideoUploadParameter param);
        /// <summary>
        /// 上传普通文件
        /// </summary>
        /// <param name="filePath"></param>
        /// <returns></returns>
        FileUploadResult UploadFile(FileUploadParameter param);
        /// <summary>
        /// 上传图片
        /// </summary>
        /// <param name="param"></param>
        /// <returns></returns>
        /// <remarks>Update:cyr(Ben) 20150317</remarks>
        ImageUploadResult UploadImage(ImageUploadParameter param);
    }

２　一批方法参数，包括了文件，图像和视频等

/// <summary>
    /// 文件上传参数基类
    /// </summary>
    public abstract class UploadParameterBase
    {
        public UploadParameterBase()
        {
            MaxSize = 1024 * 1024 * 8;
        }
        /// <summary>
        /// 前一次上传时生成的服务器端文件名，如果需要断点续传，需传入此文件名
        /// </summary>
        public string ServiceFileName { get; set; }
        /// <summary>
        /// 文件流
        /// </summary>
        public Stream Stream { get; set; }
        /// <summary>
        /// 文件名
        /// </summary>
        public string FileName { get; set; }
        /// <summary>
        /// 文件大小限制（单位bit 默认1M）
        /// </summary>
        public int MaxSize
        {
            get;
            protected set;
        }
        /// <summary>
        /// 上传文件类型限制
        /// </summary>
        public string[] FilenameExtension
        {
            get;
            set;
        }

    }      
/// <summary>
    /// 图片上传参数对象
    /// </summary>
    public class ImageUploadParameter : UploadParameterBase
    {
        /// <summary>
        /// 构造方法
        /// </summary>
        /// <param name="stream"></param>
        /// <param name="fileName"></param>
        /// <param name="filenameExtension">默认支持常用图片格式</param>
        /// <param name="maxSize"></param>
        public ImageUploadParameter(Stream stream, string fileName, string[] filenameExtension = null, int maxSize = 3)
        {
            base.Stream = stream;
            base.FileName = fileName;
            base.MaxSize = maxSize;
            base.FilenameExtension = filenameExtension ?? new string[] { ".jpeg", ".jpg", ".gif", ".png" }; ;

        }
        /// <summary>
        /// 构造方法
        /// </summary>
        /// <param name="stream"></param>
        /// <param name="fileName"></param>
        /// <param name="maxSize">单位为M</param>
        public ImageUploadParameter(Stream stream, string fileName, int maxSize)
            : this(stream, fileName, null, maxSize) { }

    }

３　一批返回类型，包括对文件，图像和视频等方法的返回数据的定义

　　/// <summary>
    /// 上传文件返回对象基类
    /// </summary>
    public abstract class UploadResultBase
    {
        /// <summary>
        /// 返回文件地址
        /// </summary>
        public string FilePath { get; set; }
        /// <summary>
        /// 错误消息列表
        /// </summary>
        public string ErrorMessage { get; set; }
        /// <summary>
        /// 是否上传成功
        /// </summary>
        public bool IsValid { get { return string.IsNullOrWhiteSpace(ErrorMessage); } }
    }      
/// <summary>
    /// 视频上传返回对象
    /// </summary>
    public class VideoUploadResult : UploadResultBase
    {
        /// <summary>
        /// 上传的视频截图地址
        /// </summary>
        public List<string> ScreenshotPaths { get; set; }
        /// <summary> 
        /// 上传状态
        /// </summary>
        public UploadStatus UploadStatus { get; set; }

        public VideoUploadResult()
        {
            ScreenshotPaths = new List<string>();
        }
        /// <summary>
        /// 把VideoPath和ScreenshotPaths拼起来  以竖线（|）隔开
        /// </summary>
        /// <returns></returns>
        public override string ToString()
        {
            StringBuilder sb = new StringBuilder();
            sb.Append(FilePath);
            foreach (var item in ScreenshotPaths)
            {
                sb.Append("|" + item);
            }
            return sb.ToString();
        }
    }

４　一个使用FastDFS实现的文件上传实现类

/// <summary>
    /// 使用fastDFS完成文件上传
    /// </summary>
    internal class FastDFSUploader : IFileUploader
    {
        /// <summary>
        /// 目录名,需要提前在fastDFS上建立
        /// </summary>
        public string DFSGroupName { get { return "tsingda"; } }
        /// <summary>
        /// FastDFS结点
        /// </summary>
        public StorageNode Node { get; private set; }
        /// <summary>
        /// 服务器地址
        /// </summary>
        public string Host { get; private set; }
        /// <summary>
        /// 失败次数
        /// </summary>
        protected int FaildCount { get; set; }

        public int MaxFaildCount { get; set; }

        public FastDFSUploader()
        {
            InitStorageNode();
            MaxFaildCount = 3;
        }

        #region Private Methods
        private void InitStorageNode()
        {
            Node = FastDFSClient.GetStorageNode(DFSGroupName);
            Host = Node.EndPoint.Address.ToString();
        }

        private List<string> CreateImagePath(string fileName)
        {
            List<string> pathList = new List<string>();
            string snapshotPath = "";
            //视频截图
            List<string> localList = new VideoSnapshoter().GetVideoSnapshots(fileName, out snapshotPath);
            foreach (var item in localList)
            {
                string aImage = SmallFileUpload(item);
                pathList.Add(aImage);
            }
            //清除本地多余的图片，有的视频截取的图片多，有的视频截取的图片少
            string[] strArr = Directory.GetFiles(snapshotPath);
            try
            {
                foreach (var strpath in strArr)
                {
                    File.Delete(strpath);
                }
                Directory.Delete(snapshotPath);
            }
            catch (Exception ex)
            {
                Logger.Core.LoggerFactory.Instance.Logger_Info("删除图片截图异常" + ex.Message);
            }
            return pathList;
        }
        private string SmallFileUpload(string filePath)
        {
            if (string.IsNullOrEmpty(filePath))
                throw new ArgumentNullException("filePath 参数不能为空");
            if (!File.Exists(filePath))
                throw new Exception("上传的文件不存在");
            byte[] content;
            using (FileStream streamUpload = new FileStream(filePath, FileMode.Open, FileAccess.Read))
            {
                using (BinaryReader reader = new BinaryReader(streamUpload))
                {
                    content = reader.ReadBytes((int)streamUpload.Length);
                }
            }
            string shortName = FastDFSClient.UploadFile(Node, content, "png");
            return GetFormatUrl(shortName);
        }
        /// <summary>
        /// 文件分块上传，适合大文件
        /// </summary>
        /// <param name="file"></param>
        /// <returns></returns>
        private string MultipartUpload(UploadParameterBase param)
        {
            Stream stream = param.Stream;
            if (stream == null)
                throw new ArgumentNullException("stream参数不能为空");
            int size = 1024 * 1024;
            byte[] content = new byte[size];
            Stream streamUpload = stream;
            //  第一个数据包上传或获取已上传的位置
            string ext = param.FileName.Substring(param.FileName.LastIndexOf('.') + 1);
            streamUpload.Read(content, 0, size);
            string shortName = FastDFSClient.UploadAppenderFile(Node, content, ext);

            BeginUploadPart(stream, shortName);

            return CompleteUpload(stream, shortName);
        }
        /// <summary>
        /// 断点续传
        /// </summary>
        /// <param name="stream"></param>
        /// <param name="serverShortName"></param>
        private void ContinueUploadPart(Stream stream, string serverShortName)
        {
            var serviceFile = FastDFSClient.GetFileInfo(Node, serverShortName);
            stream.Seek(serviceFile.FileSize, SeekOrigin.Begin);
            BeginUploadPart(stream, serverShortName);
        }
        /// <summary>
        /// 从指定位置开始上传文件
        /// </summary>
        /// <param name="stream"></param>
        /// <param name="beginOffset"></param>
        /// <param name="serverShortName"></param>
        private void BeginUploadPart(Stream stream, string serverShortName)
        {
            try
            {
                int size = 1024 * 1024;
                byte[] content = new byte[size];

                while (stream.Position < stream.Length)
                {
                    stream.Read(content, 0, size);

                    var result = FastDFSClient.AppendFile(DFSGroupName, serverShortName, content);
                    if (result.Length == 0)
                    {
                        FaildCount = 0;
                        continue;
                    }
                }
            }
            catch (Exception ex)
            {
                Logger.Core.LoggerFactory.Instance.Logger_Info("上传文件中断！" + ex.Message);
                if (NetCheck())
                {
                    //重试
                    if (FaildCount < MaxFaildCount)
                    {
                        FaildCount++;
                        InitStorageNode();
                        ContinueUploadPart(stream, serverShortName);
                    }
                    else
                    {
                        Logger.Core.LoggerFactory.Instance.Logger_Info("已达到失败重试次数仍没有上传成功"); ;
                        throw ex;
                    }
                }
                else
                {
                    Logger.Core.LoggerFactory.Instance.Logger_Info("当前网络不可用");
                    throw ex;
                }
            }
        }
        /// <summary>
        /// 网络可用为True,否则为False
        /// </summary>
        /// <returns></returns>
        private bool NetCheck()
        {
            return NetworkInterface.GetIsNetworkAvailable();
        }
        /// <summary>
        /// 拼接Url
        /// </summary>
        /// <param name="shortName"></param>
        /// <returns></returns>
        private string GetFormatUrl(string shortName)
        {
            return string.Format("http://{0}/{1}/{2}", Host, DFSGroupName, shortName);
        }

        private string CompleteUpload(Stream stream, string shortName)
        {
            stream.Close();
            return GetFormatUrl(shortName);
        }

        private string GetShortNameFromUrl(string url)
        {
            if (string.IsNullOrEmpty(url))
                return string.Empty;
            Uri uri = new Uri(url);
            string urlFirstPart = string.Format("http://{0}/{1}/", Host, DFSGroupName);
            if (!url.StartsWith(urlFirstPart))
                return string.Empty;
            return url.Substring(urlFirstPart.Length);
        }
        #endregion

        #region IFileUploader 成员

        /// <summary>
        /// 上传视频
        /// </summary>
        /// <param name="param"></param>
        /// <returns></returns>
        public VideoUploadResult UploadVideo(VideoUploadParameter param)
        {
            VideoUploadResult result = new VideoUploadResult();
            string fileName = MultipartUpload(param);
            if (param.IsScreenshot)
            {
                result.ScreenshotPaths = CreateImagePath(fileName);
            }
            result.FilePath = fileName;
            return result;
        }

        /// <summary>
        /// 上传普通文件
        /// </summary>
        /// <param name="param"></param>
        /// <returns></returns>
        public FileUploadResult UploadFile(FileUploadParameter param)
        {
            var result = new FileUploadResult();
            try
            {
                string fileName = MultipartUpload(param);
                result.FilePath = fileName;
            }
            catch (Exception ex)
            {

                result.ErrorMessage = ex.Message;
            }
            return result;
        }

        /// <summary>
        /// 上传图片
        /// </summary>
        /// <param name="param"></param>
        /// <param name="message"></param>
        /// <returns></returns>
        public ImageUploadResult UploadImage(ImageUploadParameter param)
        {
            byte[] content;
            string shortName = "";
            string ext = System.IO.Path.GetExtension(param.FileName).ToLower();
            if (param.FilenameExtension != null && param.FilenameExtension.Contains(ext))
            {
                if (param.Stream.Length > param.MaxSize)
                {
                    return new ImageUploadResult
                    {
                        ErrorMessage = "图片大小超过指定大小" + param.MaxSize / 1048576 + "M，请重新选择",
                        FilePath = shortName
                    };
                }
                else
                {
                    using (BinaryReader reader = new BinaryReader(param.Stream))
                    {
                        content = reader.ReadBytes((int)param.Stream.Length);
                    }

                    shortName = FastDFSClient.UploadFile(Node, content, ext.Contains('.') ? ext.Substring(1) : ext);
                }
            }
            else
            {
                return new ImageUploadResult
                {
                    ErrorMessage = "文件类型不匹配",
                    FilePath = shortName
                };

            }
            return new ImageUploadResult
            {
                FilePath = CompleteUpload(param.Stream, shortName),
            };
        }
        #endregion
    }

５　一个文件上传的生产者，经典的单例模式的体现

/// <summary>
    /// 文件上传生产者
    /// </summary>
    public class FileUploaderFactory
    {
        /// <summary>
        /// 上传实例
        /// </summary>
        public readonly static IFileUploader Instance;
        private static object lockObj = new object();
        static FileUploaderFactory()
        {
            if (Instance == null)
            {
                lock (lockObj)
                {
                    Instance = new FastDFSUploader();
                }
            }
        }
    }

６　前台的文件上传控件，你可以随便选择了，它与前台是解耦的，没有什么关系，用哪种方法实现都可以，呵呵

　　　　[HttpPost]
        public ActionResult Index(FormCollection form)
        {
            var file = Request.Files[0];
            var result = Project.FileUpload.FileUploaderFactory.Instance.UploadFile(new Project.FileUpload.Parameters.FileUploadParameter
            {
                FileName = file.FileName,
                Stream = file.InputStream,
            });
            ViewBag.Path = result.FilePath;
            return View();
        }

最后我们看一下我的Project.FileUpload的完整结构

它隶属于大叔的Project.Frameworks集合，我们在这里，对Project.FileUpload说一声：Hello FileUpload,We wait for you for a long time...

 回到目录

作者：仓储大叔，张占岭，

荣誉：微软MVP

QQ：853066980

支付宝扫一扫，为大叔打赏!

我心中的核心组件（可插拔的AOP）~分布式文件上传组件~基于FastDFS
上一篇: 阿里云ECS云服务器通用型g6和g5的区别以及选择建议
下一篇: 小猿日记 - 程序猿的日常日记（3）口水记小结不正经语录声明公众号