What does this debug log of Storm mean? -
i learned storm, , tested wordcount example code here. uses setdebug(true)
enable debug logging.
confused these debug information:
13867 [storm.starter.wordcounttopology.main()] info backtype.storm.daemon.nimbus - setting new assignment topology id word-count-1-1403745226: #backtype.storm.daemon.common.assignment{:master-code-dir "/tmp/4ef1358b-92ce-4051-a4ce- b33a25ace66d/nimbus/stormdist/word-count-1-1403745226", :node->host {"47fdd74b-65e0-4650- a000-f1f0418dd7c7" "ms-virtualbox"}, :executor->node+port {[2 2] ["47fdd74b-65e0-4650-a000- f1f0418dd7c7" 1027], [3 3] ["47fdd74b-65e0-4650-a000-f1f0418dd7c7" 1027], [4 4] ["47fdd74b- 65e0-4650-a000-f1f0418dd7c7" 1027], [5 5] ["47fdd74b-65e0-4650-a000-f1f0418dd7c7" 1027], [6 6] ["47fdd74b-65e0-4650-a000-f1f0418dd7c7" 1027], [7 7] ["47fdd74b-65e0-4650-a000- f1f0418dd7c7" 1027], [8 8] ["47fdd74b-65e0-4650-a000-f1f0418dd7c7" 1027], [9 9] ["47fdd74b- 65e0-4650-a000-f1f0418dd7c7" 1027], [10 10] ["47fdd74b-65e0-4650-a000-f1f0418dd7c7" 1027], [1 1] ["47fdd74b-65e0-4650-a000-f1f0418dd7c7" 1027]}, :executor->start-time-secs {[2 2] 1403745226, [3 3] 1403745226, [4 4] 1403745226, [5 5] 1403745226, [6 6] 1403745226, [7 7] 1403745226, [8 8] 1403745226, [9 9] 1403745226, [10 10] 1403745226, [1 1] 1403745226}} ... 14547 [thread-6] info backtype.storm.daemon.supervisor - launching worker assignment #backtype.storm.daemon.supervisor.localassignment{:storm-id "word-count-1- 1403745226", :executors ([2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10] [1 1])} supervisor 47fdd74b-65e0-4650-a000-f1f0418dd7c7 on port 1027 id 80043f34-85af- 4862-840e-99a58b1cf1f5 ... 16140 [thread-6] info backtype.storm.daemon.executor - loading executor count:[2 2] 16153 [thread-6] info backtype.storm.daemon.task - emitting: count __system ["startup"] 16155 [thread-6] info backtype.storm.daemon.executor - loaded executor tasks count:[2 2] 16182 [thread-6] info backtype.storm.daemon.executor - finished loading executor count:[2 2] 16186 [thread-8-count] info backtype.storm.daemon.executor - preparing bolt count:(2) 16198 [thread-6] info backtype.storm.daemon.executor - loading executor count:[3 3] 16198 [thread-6] info backtype.storm.daemon.task - emitting: count __system ["startup"] 16198 [thread-6] info backtype.storm.daemon.executor - loaded executor tasks count:[3 3] 16206 [thread-8-count] info backtype.storm.daemon.executor - prepared bolt count:(2) 16213 [thread-6] info backtype.storm.daemon.executor - finished loading executor count: [3 3] 16217 [thread-10-count] info backtype.storm.daemon.executor - preparing bolt count:(3) 16221 [thread-10-count] info backtype.storm.daemon.executor - prepared bolt count:(3)
what [1 1]
,[2 2]
... mean? can give me detailed explanation. much.
thanks comment, found useful there.
after taking deep @ concepts of component
, node
, worker
, executor
, task
, know [1 1]
means.
firstly
:node->host {"47fdd74b-65e0-4650-a000-f1f0418dd7c7" "ms-virtualbox"}
this means there's 1 node here. node id 47fdd74b-65e0-4650-a000-f1f0418dd7c7
, responding host machine ms-virtualbox
.
secondly
:executor->node+port {[2 2] ["47fdd74b-65e0-4650-a000-f1f0418dd7c7" 1027], ...
the executor thread runs spout/bolt instance, above sentence shows executor is. [2 2] executor id, responds node 47fdd74b-65e0-4650-a000-f1f0418dd7c7
's 1027
port.
since wordcounttopology
has code snippet conf.setmaxtaskparallelism(3);
, , there 3 spout/bolt, maximum number of executors spout/bolt 9 (from [2 2]
[10 10]
). (1 task responds 1 executor in case) in addition, executor ([1 1]
) spawned called acker
, track tuple trees , detect when spout tuple has been processed. so, there 10 executors totally, [1 1]
[10 10]
.
- useful references:
understanding parallelism of storm topology
Comments
Post a Comment