Salut à tous !

Je suis bloqué depuis ce matin sur le premier exercice du cours d'OpenClassrooms sur Hadoop, j'ai vraiment cherché partout...

Tout comme dans l'exercice, je pose sur mon HDFS le fichier de test lejourseleve.txt dans un répertoire /input que j'ai créé dessus, je compile mes source, génère le jar et lance hadoop avec

hadoop jar ooc_cours1_wordcount.jar ooc.cours1.wordcount.WordCountDriver /input/lejourseleve.txt /result

Et j'obtiens le résultat suivant :

17/10/20 17:26:09 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/10/20 17:26:10 INFO input.FileInputFormat: Total input paths to process : 1
17/10/20 17:26:10 INFO mapreduce.JobSubmitter: number of splits:1
17/10/20 17:26:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1508513080432_0002
17/10/20 17:26:10 INFO impl.YarnClientImpl: Submitted application application_1508513080432_0002
17/10/20 17:26:10 INFO mapreduce.Job: The url to track the job: http://de-HP-ENVY-15-Notebook-PC:8088/proxy/application_1508513080432_0002/
17/10/20 17:26:10 INFO mapreduce.Job: Running job: job_1508513080432_0002

Ca reste bloqué à cette étape infiniment, et je peux voir via l'interface web que le MapReduce est resté bloqué à l'état ACCEPTED.

Voir ici l'état de mon cluster

Voir ici les détais sur le job concerné

Peu importe ce que j'essaie, toujours le même état : "ACCEPTED: waiting for AM container to be allocated, launched and register with RM."

Les conseils qui reviennent le plus souvent sur StackOverflow se résument à "tu n'as pas assez d'espace disque, libères-en", sauf qu'il me reste plus de 50Go, donc non.

Les conseils qui reviennent le plus après, que ce soit sur SO ou sur Github, c'est "tu n'alloues pas correctement tes ressources, modifies tel ou tel fichier de config" (les .xml situés dans etc/hadoop)

J'essaie tout ce qu'on me propose, redémarre mon Hadoop, mais rien n'y fait, toujours le même résultat..

Si quelqu'un a une idée et peut m'aider ça me rendrai vraiment service, merci d'avance !

Au cas où vous avez besoin d'infos complémentaires :

Les fichiers source .java sont WordCountMapper, WordCountReducer et WordCountDriver du cours, inchangés

Les fichiers de config que j'ai modifié :

- capacity-scheduler.xml

<property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare 
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare 
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>100</value>
    <description>Default queue target capacity.</description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler 
      attempts to schedule rack-local containers. 
      Typically this should be set to number of nodes in the cluster, By default is setting 
      approximately number of nodes in one rack which is 40.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
    <description>
      A list of mappings that will be used to assign jobs to queues
      The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
      Typically this list will be used to map users to queues,
      for example, u:%user:%user maps all users to queues with the same name
      as the user.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
    </description>
  </property>

- core-site.xml

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>

- hdfs-site.xml

<property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

- mapred-site.xml

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

	<property>  
	    <name>yarn.app.mapreduce.am.resource.mb</name>  
	    <value>1228</value>
	</property>
	<property> 
	    <name>yarn.app.mapreduce.am.command-opts</name> 
	    <value>-Xmx983m</value>
	</property>
	<property>
	    <name>mapreduce.map.memory.mb</name>
	    <value>1228</value>
	</property>
	<property>
	    <name>mapreduce.reduce.memory.mb</name>
	    <value>1228</value>
	</property>
	<property>
	    <name>mapreduce.map.java.opts</name>
	    <value>-Xmx983m</value>
	</property>
	<property>
	    <name>mapreduce.reduce.java.opts</name>
	    <value>-Xmx983m</value>
	</property>

- yarn-site.xml

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
	
	<property> 
	    <name>yarn.scheduler.minimum-allocation-mb</name> 
	    <value>1228</value>
	</property>
	<property> 
	    <name>yarn.scheduler.maximum-allocation-mb</name> 
	    <value>9830</value>
	</property>
	<property> 
	    <name>yarn.nodemanager.resource.memory-mb</name> 
	    <value>9830</value>
	</property>

-
Edité par Dead End 20 octobre 2017 à 18:21:57