1. Flink cluster architecture
- Job Manager: 管理节点，每个集群至少一个，管理整个集群计算资源，Job管理与调度执行，以及Checkpoint协调
- Task Manager: 每个集群有很多Task Manager， 负责计算资源提供
- Client: 本地执行应用main()方法解析Job Graph对象，病最终将Job Gragh提交到Job Manager运行，同时监控job执行的状态
2. Job Manager
- Checkpoint Coordinator
- JobGraph => Execution Graph
- Task deploy and dispatch
- RPC communication(Actor System)
- Job acceptance(Job Dispatch)
- 集群资源管理(ResourceManager)
- TaskManager注册与管理
3. Task Manger
- Task Execution
- Network Manager
- Shuffle Environment Management
- RPC Communication(Actor system)
- Heartbeat with Job Manager and Resource Manager
- Data Exchange
- Memory Management
- Register to Resource Management
- Offer slots to Job Manager
4. Client
- Application’s main() Method 执行
- Job Graph Generate
- Execution Environment 管理
- Job 提交与运行
- Dependency jar ship
- RPC with Job Manager
- 集群部署(Cluster Deploy)
5. Job Graph
- 通过DAG方式表达用户程序
- 不通接口程序的抽象表达
- 客户端和集群之前的job描述载体
- 节点(Vertices), Result参数
- Flink1.11之前只能在Client中生成