Now in the occupation of the Executor . Each application has its own executors . The executor interface definition is quite simple:.
Each application has its own executor process . Executor 只会调用resolve或reject。The executor should call only one resolve or reject.Each application has its own executor process . In terms of the analogy above: the executor is the“singer”. 一个Spark集群一般拥有单个的Driver和多个的Executor 。 Spark consists of a single driver and multiple executors . 会自动并立即调用executor (通过newPromise)。 The executor is called automatically and immediately(by the new Promise). 传递给newPromise的函数称之为executor 。 The function passed to new Promise is called the executor .应用程序可以使用Executor 框架来创建线程池. An application can use the Executor framework, in order to create a thread pool. 基本上,executor 代替了显式创建和管理线程。 Basically, the executor replaces the explicit creation and management of a thread. Executor 接收两个函数对象作为参数:resolve()和reject()。The executor is passed two functions named resolve() and reject() as arguments. 经过一秒钟的思考后,executor 调用resolve("done")来产生结果:. After one second of“processing” the executor calls resolve("done") to produce the result:. Executor 负责执行work,以task的形式,与您存储任何您缓存的数据一样。The executors are responsible for performing work, in the form of tasks, as well as for storing any data that you cache. ApplicationMaster只负责从YARN中请求executor container。 The ApplicationMaster is responsible only for requesting executor containers from YARN. 这个包中包含了Executor Framework,这是一个很灵活的基于接口的任务执行工具。 This package contains an Executor Framework, which is a flexible interface-based task execution facility. 当应用变得空闲时,它的executor 会释放资源并且能够被其它的应用获得。 When the application becomes idle, its executors are released and can be acquired by other applications. 接下来,它将发送您的应用代码(通过JAR或者Python文件定义传递给SparkContext)至Executor 。 Next, it sends your application code(defined by JAR or Python files passed to SparkContext) to the executors . 请注意,JAR和文件被复制到executor 节点上每个SparkContext的工作目录下。 Note that JARs and files are copied to the working directory for each SparkContext on the executor nodes. 在JavaExecutor 框架自身实现了生产者消费者模式它们分别负责添加和执行任务。 In Java library, Executor framework implements Producer Consumer design pattern to separate responsibility of addition and execution of task. Worker节点故障:任何运行executor 的worker节点都有可能出故障,那样在这个节点中的所有内存数据都会丢失。 Failure of a Worker Node- Any of the worker nodes running executors can fail, and all in-memory data on those nodes will be lost. 实际上,executor 通常会异步执行一些动作,然后在一段时间后调用resolve/reject,但它不必那么做。 In practice, an executor usually does something asynchronously and calls resolve/reject after some time, but it doesn't have to. 每个driver都有一个WebUI,通常在端口4040上,可以显示有关正在运行的task,executor ,和存储使用情况的信息。 Each driver program has a web UI, typically on port 4040, that displays information about running tasks, executors , and storage usage. Java提供了Executor 接口来执行线程池中的任务,提供ExecutorServive接口来管理和控制任务。 Java provides the Executor interface for executing tasks in a thread pool and the ExecutorService interface for managing and controlling tasks. 比如broadcast变量需要为每个executor 复制一遍,这么多小executor 会导致更多的数据拷贝。 For example, broadcast variables must be replicated once on each executor , so many small executors results in many more copies of the data. 比如broadcast变量需要为每个executor 复制一遍,这么多小executor 会导致更多的数据拷贝。 For example, broadcast variables need to be replicated once on each executor , so many small executors will result in many more copies of the data.
Display more examples
Results: 29 ,
Time: 0.0223