天天看點

django 資料處理OSI七層網絡模型wsgi協定中間件資料處理

在上面提到了django内部的資料流轉,那麼就順便說說request在django内部流轉,最終生成response。下面這幅圖就很好說明了資料處理過程。

django 資料處理OSI七層網絡模型wsgi協定中間件資料處理

1. 使用者在浏覽器輸入url時,浏覽器會生成請求封包通過 OSI七層網絡模型 發給服務

2. 請求到達伺服器時,先通過 wsgi協定 将http請求資料轉化為python請求對象

3. 請求到達request 中間件,中間件對request請求做預處理檢查,如果檢查不通過則直接傳回 response,否則進入下一步

4. 到達django路由系統,找到對應的視圖函數

5. 視圖函數通過ORM通路底層資料,并進行相應處理傳回response

OSI七層網絡模型

django 資料處理OSI七層網絡模型wsgi協定中間件資料處理

當使用者在浏覽器輸入網址時,浏覽器會将資料封裝成請求資料包括請求頭和請求體(應用層),經過轉換、壓縮、加密,請求資料被轉化成 message封包(表示層),在伺服器和用戶端建立會話連接配接(會話層)。

(傳輸層)解封message封包後得到segment資料段,根據tcp頭或udp頭(源端口和目的端口,傳輸資料序号)進行傳輸。(網絡層)解封上層資料後得到packet網絡包,根據目的ip和源ip進行路由轉發。在每段路由轉發過程,(資料鍊路層)資料包繼續被解封成frame資料幀,并根據以太網資料進行節點對節點之間的傳輸。此時,(實體層)資料将會以bit比特流方式傳輸。

到達目的位址後,資料将會像上述逆序封裝。最終伺服器收到request請求資料

小知識點:

非官方語言,僅小故事記憶

1. 三步握手建立連接配接

django 資料處理OSI七層網絡模型wsgi協定中間件資料處理

某天,A想打電話給B,但又怕影響到B工作。于是,

  1. A發了條SYN短信給B:我想打電話給你,可以嗎
  2. B回複ACK資訊:剛才的資訊已經收到了,并約定SYN通信
  3. A回複ACK資訊:收到,到時間聯系。

2. 四步揮手斷開連接配接

django 資料處理OSI七層網絡模型wsgi協定中間件資料處理

兩人聊了一段時間,A想挂斷電話。

  1. A發送FIN:我沒有要說的了,挂斷吧
  2. B發送ACK:我知道了,我還有幾句要說的
  3. B巴拉巴拉說了一通後,回複FIN、ACK:好了,我們結束通話吧
  4. A回複ACK:好,我挂斷了

wsgi協定

WSGI協定(web server gateway interface)是一種通信協定,主要包括兩部分:

  1. server: 從用戶端接收請求,将request轉發給application,将application傳回的response傳回給用戶端
  2. application: 接收由server轉發的request,并将處理結果傳回給server

uWSGI和gunicorn1是實作了wsgi server協定的伺服器,django,flask是實作了wsgi application協定的web架構, 可以根據項目實際情況搭配使用。

那麼,我就以gunicorn+django為例說明:

當我們建立django項目時,

python manage.py startproject HelloWorld
           

可以檢視項目目錄

$ tree
| -- HelloWorld
|     | -- __init__.py
|     | -- settings.py 
|     | -- urls.py
|     | -- wsgi.py
| -- manage.py 
           

啟動django項目時,執行指令

gunicorn HelloWorld.wsgi
           

此時,gunicorn 建立起 WSGI 伺服器,并與HelloWorld.wsgi application建立連結。

application 源碼解析

class WSGIHandler(base.BaseHandler):
    request_class = WSGIRequest

    def __init__(self, *args, **kwargs):
        super(WSGIHandler, self).__init__(*args, **kwargs)
        # 加載中間件
        self.load_middleware()

    def __call__(self, environ, start_response):
    	# 設定環境變量資訊,如DJANGO_SETTINGS_MODULE
        set_script_prefix(get_script_name(environ))
        # 請求處理前發送資訊
        signals.request_started.send(sender=self.__class__, environ=environ)
        # 初始化request 對象
        request = self.request_class(environ)
        # 處理request,生成response(通過中間件處理)
        response = self.get_response(request)

        response._handler_class = self.__class__
		
		# response資料處理
        status = '%d %s' % (response.status_code, response.reason_phrase)
        response_headers = [(str(k), str(v)) for k, v in response.items()]
        for c in response.cookies.values():
            response_headers.append((str('Set-Cookie'), str(c.output(header=''))))
        # server提供的回調方法,将響應的header和status傳回給server
        start_response(force_str(status), response_headers)
        if getattr(response, 'file_to_stream', None) is not None and environ.get('wsgi.file_wrapper'):
            response = environ['wsgi.file_wrapper'](response.file_to_stream)
        return response

           
  • 雖然django本身也提供了runserver來啟動wsgi server,但這種啟動方式是單程序的。我更推薦使用了uwsgi\gunicorn\Apache等方式啟動服務,實作使用者請求資料處理和應用業務邏輯處理隔離開。同時,uwsgi\gunicorn\Apache還有高并發,高性能,多程序處理的優勢。

中間件資料處理

中間件是介于request處理和response傳回之間的一道處理過程,将請求資料處理和業務邏輯分離。

源碼分析

在加載中間件load_middleware時,我們可以看到下述代碼:

if hasattr(mw_instance, 'process_request'):
                    self._request_middleware.append(mw_instance.process_request)
                if hasattr(mw_instance, 'process_view'):
                    self._view_middleware.append(mw_instance.process_view)
                if hasattr(mw_instance, 'process_template_response'):
                    self._template_response_middleware.insert(0, mw_instance.process_template_response)
                if hasattr(mw_instance, 'process_response'):
                    self._response_middleware.insert(0, mw_instance.process_response)
                if hasattr(mw_instance, 'process_exception'):
                    self._exception_middleware.insert(0, mw_instance.process_exception)
           

這意味着中間件可以預定義5個函數: process_request 請求預處理、process_view 視圖預處理、process_template_response 模闆渲染處理、process_response 響應處理、 process_exception 異常處理

在這裡要注意: 前兩個請求函數順序處理,後三個響應函數逆序處理。

def get_response(self, request):
        urlconf = settings.ROOT_URLCONF
        urlresolvers.set_urlconf(urlconf)
        resolver = urlresolvers.get_resolver(urlconf)

        response_is_rendered = False
        try:
            response = None
			# 執行process_request函數,若傳回response直接中斷後續中間件的處理
            for middleware_method in self._request_middleware:
                response = middleware_method(request)
                if response:
                    break
			
            if response is None:
                if hasattr(request, 'urlconf'):
                    urlconf = request.urlconf
                    urlresolvers.set_urlconf(urlconf)
                    resolver = urlresolvers.get_resolver(urlconf)
				# 處理url請求
                resolver_match = resolver.resolve(request.path_info)
                # callback: 處理request的python函數  callback_args: 位置參數   callback_kwargs: 關鍵字參數字典
                callback, callback_args, callback_kwargs = resolver_match
                request.resolver_match = resolver_match

                # 執行 process_view 函數,同樣傳回response,則中斷後續中間件處理
                for middleware_method in self._view_middleware:
                    response = middleware_method(request, callback, callback_args, callback_kwargs)
                    if response:
                        break
			
			# 進入 callback 函數,執行業務邏輯;執行報錯進入process_exception函數處理
            if response is None:
                wrapped_callback = self.make_view_atomic(callback)
                try:
                    response = wrapped_callback(request, *callback_args, **callback_kwargs)
                except Exception as e:
                    response = self.process_exception_by_middleware(e, request)
			
            if response is None:
                if isinstance(callback, types.FunctionType):    # FBV
                    view_name = callback.__name__
                else:                                           # CBV
                    view_name = callback.__class__.__name__ + '.__call__'
                raise ValueError("The view %s.%s didn't return an HttpResponse object. It returned None instead."
                                 % (callback.__module__, view_name))
	        # 當response對象有 render 方法時,才會調用 
            if hasattr(response, 'render') and callable(response.render):
                for middleware_method in self._template_response_middleware:
                    response = middleware_method(request, response)
                    # Complain if the template response middleware returned None (a common error).
                    if response is None:
                        raise ValueError(
                            "%s.process_template_response didn't return an "
                            "HttpResponse object. It returned None instead."
                            % (middleware_method.__self__.__class__.__name__))
                # response 渲染處理
                try:
                    response = response.render()
                except Exception as e:
                    response = self.process_exception_by_middleware(e, request)

                response_is_rendered = True

        except http.Http404 as exc:
            logger.warning('Not Found: %s', request.path,
                        extra={
                            'status_code': 404,
                            'request': request
                        })
            if settings.DEBUG:
                response = debug.technical_404_response(request, exc)
            else:
                response = self.get_exception_response(request, resolver, 404, exc)

        except PermissionDenied as exc:
            logger.warning(
                'Forbidden (Permission denied): %s', request.path,
                extra={
                    'status_code': 403,
                    'request': request
                })
            response = self.get_exception_response(request, resolver, 403, exc)

        except MultiPartParserError as exc:
            logger.warning(
                'Bad request (Unable to parse request body): %s', request.path,
                extra={
                    'status_code': 400,
                    'request': request
                })
            response = self.get_exception_response(request, resolver, 400, exc)

        except SuspiciousOperation as exc:
            # The request logger receives events for any problematic request
            # The security logger receives events for all SuspiciousOperations
            security_logger = logging.getLogger('django.security.%s' %
                            exc.__class__.__name__)
            security_logger.error(
                force_text(exc),
                extra={
                    'status_code': 400,
                    'request': request
                })
            if settings.DEBUG:
                return debug.technical_500_response(request, *sys.exc_info(), status_code=400)

            response = self.get_exception_response(request, resolver, 400, exc)

        except SystemExit:
            # Allow sys.exit() to actually exit. See tickets #1023 and #4701
            raise

        except:  # Handle everything else.
            # Get the exception info now, in case another exception is thrown later.
            signals.got_request_exception.send(sender=self.__class__, request=request)
            response = self.handle_uncaught_exception(request, resolver, sys.exc_info())

        try:
           # 執行 process_response 函數
            for middleware_method in self._response_middleware:
                response = middleware_method(request, response)
                # Complain if the response middleware returned None (a common error).
                if response is None:
                    raise ValueError(
                        "%s.process_response didn't return an "
                        "HttpResponse object. It returned None instead."
                        % (middleware_method.__self__.__class__.__name__))
            response = self.apply_response_fixes(request, response)
        except: 
            signals.got_request_exception.send(sender=self.__class__, request=request)
            response = self.handle_uncaught_exception(request, resolver, sys.exc_info())

        response._closable_objects.append(request)
        if not response_is_rendered and callable(getattr(response, 'render', None)):
            response = response.render()
		# 傳回 response 對象
        return response
           

總結

  1. gunicorn較uwsgi配置簡單,易上手,且響應速度快。但uwsgi功能強大,穩定性強,吞吐量大 ↩︎