Thursday, September 26, 2013

Hadoop: Exit reduce function for the rest of input on some condition

Say we have a reduce function where in we want to halt the reduce function after processing some 'n' keys. We have set a counter to increment on each key, and on condition being satisfied return from the reduce function without bothering about rest of the keys.

Fortunately with the new mapreduce API, we have the ability to do that now.

You can achieve this by overriding the run() of the Reducer class.

Let's straight get to the code:

public static class Reduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable> {

  //reduce method here

  // Override the run()
  @override
  public void run(Context context) throws IOException, InterruptedException {
    setup(context);
    int count = 0;
    while (context.nextKey()) {
        if (count++ < n) {
        reduce(context.getCurrentKey(), context.getValues(), context);
        } else {
            // exit or do whatever you want
        }
    }
    cleanup(context);
  }
}

43 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. This comment has been removed by a blog administrator.

    ReplyDelete
  4. This comment has been removed by a blog administrator.

    ReplyDelete
  5. This comment has been removed by a blog administrator.

    ReplyDelete

  6. Your posts is really helpful for me.Thanks for your wonderful post. I am very happy to read your post.very nice !!!VMWare Training in chennai | VMWare Training chennai | VMWare course in chennai

    ReplyDelete
  7. This is extremely helpful info!! Very good work. Everything is very interesting to learn and easy to understood. Thank you for giving information. Cloud Computing Training in chennai | Cloud Computing Training chennai | Cloud Computing Course in chennai | Cloud Computing Course chennai

    ReplyDelete
  8. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this.

    ReplyDelete
  9. Truely a very good article on how to handle the future technology. This content creates a new hope and inspiration within me. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks :)

    Software testing training in chennai | Software testing training | Software testing training chennai

    ReplyDelete
  10. Using big data analytics may give the companies many fruitful results, the findings can be implemented in their business decisions so as to minimize their risk and to cut the costs.
    hadoop training in chennai|big data training|big data training in chennai

    ReplyDelete
  11. Cloud computing is the next big thing, through cloud the users have the liberty to use a shared network. The companies can focus on core business parts rather than investing heavily on infrastucture.
    cloud computing training in chennai|cloud computing courses in chennai|cloud computing training

    ReplyDelete
  12. Big data is used extensively in MNC today as using big data leads to accurate decision making and there are is a huge demand for the big data analysts.
    Big data training in Chennai | Hadoop training in Chennai | Big data training institute in Chennai

    ReplyDelete
  13. Much obliged to you for requiring significant investment to give us a portion of the valuable and restrictive data with us.
    Regards,
    VMware Training in chennai | Vmware Training center in Chennai |

    ReplyDelete
  14. The blog gave me idea about hadoop exit reduce function Thanks for sharing it
    Hadoop Training in Chennai

    ReplyDelete
  15. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post.R Programming Online Training | Hadoop Online Training

    ReplyDelete
  16. Great sharing.. Looking for Hadoop testing from professional experts with placement support reach us Hadoop Training in Chennai

    ReplyDelete
  17. hi welcome to this blog. really you have post an informative blog. it will be really helpful to many peoples. thank you for sharing this blog.
    python training in chennai

    ReplyDelete
  18. Hi admin.thank you for sharing that blog.Huge Data implies BIG Opportunities with Big bucks. Come snatch them with both hands!....
    Dot Net Training in Chennai
    Android Training in Chennai
    Hadoop Training in Chennai
    Selenium Training in Chennai

    ReplyDelete
  19. Nowadays Hadoop marketing and job opening were high.Any of college student interesting studied for Hadoop. Leads to accurate decision Hadoop data analytics marketing and there are is a huge demand for the big data analysts.
    Hadoop Online Training
    Learn Hadoop Training Online
    Online Hadoop Training

    ReplyDelete
  20. Good Knowledge sharing about Big Data .
    Big Data has a huge demand in IT Industry.
    http://eonlinetraining.co/course/big-data-hadoop-online-training/

    ReplyDelete
  21. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…

    Hadoop Training In Chennai

    ReplyDelete
  22. Much thanks to you such a great amount for offering this value capable substance to us. The idea taken here will be helpful for my future projects and i will most likely actualize them in my review. Continue blogging article this way.
    Hadoop Training in Chennai |
    Dot Net Training in Chennai

    ReplyDelete
  23. Everyone wants to get unique place in the IT industry’s for that you need to upgrade your skills, your blog helps me improvise my skill set to get good career, keep sharing your thoughts with us.
    Hadoop Online Training

    ReplyDelete
  24. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post.


    Hadoop Training in Bangalore

    ReplyDelete
  25. really nice blog has been shared by you. before i read this blog i did n't have any knowledge about this but now i got some knowledge so keep on sharing such kind of an interesting blogs.
    hadoop training in Chennai

    ReplyDelete
  26. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information.
    Hadoop Training in Chennai

    ReplyDelete
  27. Everyone wants to get unique place in the IT industry’s for that you need to upgrade your skills, your blog helps me improvise my skill set to get good career, keep sharing your thoughts with us.


    Hadoop Training in Marathahalli|
    Hadoop Training in Bangalore|
    Data science training in Marathahalli|
    Data science training in Bangalore|

    ReplyDelete
  28. Hello Admin,
    I really enjoyed while reading your article, the information you have mentioned in this post was damn good. Keep sharing your blog with updated and useful information.
    Regards,
    Hadoop Training in Chennai

    ReplyDelete
  29. Nice blog about Hadoop and the way of explaining about exit reduce is really good. Hadoop Training Institute in Chennai

    ReplyDelete
  30. Thanks a lot very much for the high quality and results-oriented help. I won’t think twice to endorse your blog post to anybody who wants and needs support about this area. AWS Training in Bangalore

    ReplyDelete
  31. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.

    Aws training in Bangalore

    ReplyDelete
  32. I am commenting to let you know what a terrific experience I enjoyed reading through your web page. I noticed a wide variety of pieces, with the inclusion of what it is like to have an awesome helping style to have the rest without hassle grasp some grueling matters
    Hadoop Training in Bangalore

    ReplyDelete
  33. I wish to indicate because of you only to bail me out of this specific trouble.As a consequence of checking through the net and meeting systems that were not beneficial, I thought my life was finished.

    Java Training in Bangalore|

    ReplyDelete
  34. This comment has been removed by the author.

    ReplyDelete
  35. I and my friends were going through the nice, helpful tips from the blog then the sudden came up with an awful suspicion I never expressed respect to the website owner for those secrets.
    digital training in chennai

    ReplyDelete
  36. I‘d mention that most of us visitors are endowed to exist in a fabulous place with very many wonderful individuals with very helpful things.
    hadoop training in bangalore
    hadoop training in chennai

    ReplyDelete
  37. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here.

    Data Science Training in Bangalore

    Datascience Training in Chennai

    ReplyDelete

Any feedback, good or bad is most welcome.

Name

Email *

Message *