[Spark][Python][Application]非互動式運作Spark Application 的例子

非互動式運作Spark Application 的例子

$ cat Count.py

import sys

from pyspark import SparkContext

if __name__ == "__main__":

sc = SparkContext()

logfile = sys.argv[1]

count = sc.textFile(logfile).filter(lambda line: '.jpg' in line).count()

print "JPG requests: ", count

sc.stop()

$

$ spark-submit --master yarn-client Count.py /test/weblogs/*

Number of JPG requests: 10258

本文轉自健哥的資料花園部落格園部落格，原文連結：http://www.cnblogs.com/gaojian/p/7749427.html，如需轉載請自行聯系原作者