Wednesday, June 11, 2014

What is the need to use job.setJarByClass in the driver section of Hadoop MapReduce solution?

Have you ever thought of this?

We do provide the jar to be executed by hadoop while executing hadoop command, don't we?

$ hadoop jar /some-jar.jar

So, why do we need to have the following line of code in our Driver section while declaring the Job object properties:
job.setJarByClass(SomeClass.class);

Answer to that is very simple. Here you help Hadoop to find out that which jar it should send to nodes to perform Map and Reduce tasks. Your some-jar.jar might have various other jars in it's classpath, also your driver code might be in a separate jar than that of your Mapper and Reducer classes.

Hence, using this setJarByClass method we tell Hadoop to find out the relevant jar by finding out that the class specified as it's parameter to be present as part of that jar. So usually you should provide either MapperImplementation.class or your Reducer implementation or any other class which is present in the same jar as that pf Mapper and Reducer. Also make sure that both Mapper and Reducer are part of the same jar.

21 comments:

  1. What if I don't specify it. I mean if I comment it off?

    ReplyDelete
  2. Great post. I really learned some new things in here. So thanks.

    catering services in chennai

    ReplyDelete
    Replies
    1. I have read your blog its very attractive and impressive. I like it your blog.

      Java Training in Chennai Core Java Training in Chennai Core Java Training in Chennai

      Java Online Training Java Online Training Core Java 8 Training in Chennai Core java 8 online training JavaEE Training in Chennai Java EE Training in Chennai

      Delete
    2. Hibernate Online Training Hibernate Online Training Hibernate Training in Chennai Hibernate Training in Chennai Java Online Training Java Online Training

      Hibernate Training Institutes in ChennaiHibernate Training Institutes in Chennai Java Online Training Java Online Training Java Online Training Java Online Training

      Delete
    3. Java Training Institutes Java Training Institutes Java EE Training in Chennai Java EE Training in Chennai Java Spring Hibernate Training Institutes in Chennai J2EE Training Institutes in Chennai J2EE Training Institutes in Chennai Core Java Training Institutes in Chennai Core Java Training Institutes in Chennai

      Delete
  3. IVR is a system that reply to human when the admin in not Available.... Human Sleep but IVR don't sleep.......

    IVR Application

    ReplyDelete
  4. My Arcus offer java training with 100% placement. Our java training course that includes fundamentals and advance java training program with high priority jobs. java j2ee training with placement having more exposure in most of the industry nowadays in depth manner of java .

    java training in chennai

    ReplyDelete
  5. Hi this is very informative and useful blog for all kinda peoples.

    PHP Training in Chennai

    ReplyDelete
  6. This is truly a standout amongst the most gainful online journals I've ever skimmed on this subject. I am exceptionally happy to peruse such an incredible blog and thank you for offering this great information to us.
    Spring Training in Chennai | Spring Hibernate Training | Core Spring Training

    ReplyDelete
  7. Nice post, Thanks for sharing this useful information with us.
    IT Job Training in Chennai

    ReplyDelete
  8. This post was quite awesome and interesting to read. Congrats for your work. Thanks a lot for providing this with us. Keep on updating this with us regularly.

    ReplyDelete
  9. This post was quite awesome and interesting to read. Congrats for your work. Thanks a lot for providing this with us. Keep on updating this with us regularly.

    ReplyDelete
  10. ERPTREE is a leading oracle fuison HCM Training Institute. we offer this course through online we have great experience in succeeding students through online courses. we can calculate our performance through their honest comments in our sites in supporting our services. we have referral program so candidates can earn money through referral. you can share your live experience with other can generate you some money.



    Oracle Fusion Cloud HCM Training

    ReplyDelete
  11. Nice and good informative article, thanks for sharing your valuable ideas, keep rocks.
    Software Testing Training in chennai | Software Testing Training in velachery

    ReplyDelete
  12. As an engineer by training, I had concentrated on effectiveness of process. In summary the job seeking system gets the job seeker ahead of the competition and closer to the hiring manager, effectively what a good recruiter does to win recruitment business. guarantor

    ReplyDelete
  13. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here.

    Java Training in Bangalore

    ReplyDelete

Any feedback, good or bad is most welcome.

Name

Email *

Message *