Ask Computer Engineering Expert

Mining Big Data Assignment: Basics and Map-Reduce

Exercise 1 - Suspected Pairs

Using the information from the first lecture (or Section 1.2.3 in the textbook), what would be the number of suspected pairs if the following changes were made to the data (all changes should be applied at once).

1. The number of days of observation was raised to 5000.

2. The number of people observed was raised to 5 billion (and there were therefore 500,000 hotels).

3. We only reported a pair as suspect if they were at the same hotel at the same time on four different days.

Exercise 2 - Hadoop

For this exercise, you have to set up and configure your system to use Hadoop. Follow the instructions in Stanford document (in attached file) and set up the virtual machine as described in Section 1. Run the example program of Section 2 and carry out the different steps given in that section.

  • Write your own Hadoop Map-Reduce job that outputs the number of words that start with each letter (see Sections 2.5 and 3 of the Stanford document).
  • Run your job on the file in standalone mode and pseudo-distributed mode and record the output.

Exercise 3 - Friend Recommendation System (Stanford)

Write a MapReduce program in Hadoop that implements a simple People You Might Know social network friendship recommendation algorithm. The key idea is that if two people have a lot of mutual friends, then the system should recommend that they connect with each other. You have to run the program on the system setup in Exercise 2 in order to receive points for this exercise.

Input: Download the input file, the input file contains the adjacency list and has multiple lines in the following format:

Here, is a unique integer ID corresponding to a unique user and is a comma separated list of unique IDs corresponding to the friends of the user with the unique ID . Note that the friendships are mutual (i.e., edges are undirected): if A is friend with B then B is also friend with A. Algorithm: Let us use a simple algorithm such that, for each user U, the algorithm recommends N = 10 users who are not already friends with U, but have the most number of mutual friends in common with U.

Output: The output should contain one line per user in the following format:

where is a unique ID corresponding to a user and is a comma separated list of unique IDs corresponding to the algorithms recommendation of people that might know, ordered in decreasing number of mutual friends. Even if a user has less than 10 second-degree friends, output all of them in decreasing order of the number of mutual friends. If there are recommended users with the same number of mutual friends, then output those user IDs in numerically ascending order. Also, please provide a description of how you are going to use MapReduce jobs to solve this problem. Do not write more than 3 to 4 sentences for this: we only want a very high-level description of your strategy to tackle this problem. Note: It is possible to solve this question with a single MapReduce job. But if your solution requires multiple map reduce jobs, then that is fine too.

For your submission

  • Include your source code
  • Include in your writeup a short paragraph describing your algorithm to tackle this problem.
  • Include in your writeup the recommendations for the users with following user IDs: 924, 8941, 8942, 9019, 9020, 9021, 9022, 9990, 9992, 9993.

Exercise 4 - MapReduce

This exercise has 4 parts. In this exercise, you will be writing and implementing two MapReduce programs. Both are a bit challenging, but they will help you to have a better understanding about the MapReduce implementation. After you write the programs, you will need to answer some questions about them.

Remember that neither problem is case sensitive, so transform words to lowercase or uppercase. Also remember to use the StringTokenizer to find the correct answers.

Part 1: Write a program that processes the FirstInputFile and the SecondInputFile (attached). This program should count the number of words with a specific amount of letters in these files - for example, the number of words with 4 letters, 5 letters and so on. If one word is repeated 20 times in the text, count it individually 20 times.

Part 2: Answer Questions 1-6.

Q1: How many words are there with length 10 in FirstInputFile?

Q2: How many words are there with length 4 in FirstInputFile?

Q3: What is the longest length between words and what is its frequency in FirstInputFile?

Q4: How many words are there with length 2 in SecondInputFile?

Q5: How many words are there with length 5 in SecondInputFile?

Q6: What is the most frequent length and what is its frequency in SecondInputFile?

Part 3: Write a second program that again processes the FirstInputFile (attached) and the SecondInputFile (attached). However, in addition to counting the number of words with a specific amount of letters, if one word is repeated several times, count it only once. So, your output should be the frequency of words with same length, but count a repeated word only once. Note: You may need to use 2 MapReduce jobs.

Part 4: Answer Questions 7-12.

Q7: How many words are there with length 10 in FirstInputFile?

Q8: How many words are there with length 4 in FirstInputFile?

Q9: What is the most frequent length and what is its frequency in FirstInputFile?

Q10: How many words are there with length 5 in SecondInputFile?

Q11: How many words are there with length 2 in SecondInputFile?

Q12: What is the second-most frequent length and what is its frequency in SecondInputFile?

Exercise 5 - Summary of 2.4 and 2.5

For this exercise you have to read Section 2.3.9-2.3.11, 2.4, and 2.5 in Leskovec, Rajaraman, Ullman (second edition, 2014).

  • Summarize the content of 2.4 in your own words (600 words).
  • Summarize the content of 2.5 in your own words (600 words).

Attachment:- Assignment File.rar

Computer Engineering, Engineering

  • Category:- Computer Engineering
  • Reference No.:- M92749381

Have any Question?


Related Questions in Computer Engineering

Does bmw have a guided missile corporate culture and

Does BMW have a guided missile corporate culture, and incubator corporate culture, a family corporate culture, or an Eiffel tower corporate culture?

Rebecca borrows 10000 at 18 compounded annually she pays

Rebecca borrows $10,000 at 18% compounded annually. She pays off the loan over a 5-year period with annual payments, starting at year 1. Each successive payment is $700 greater than the previous payment. (a) How much was ...

Jeff decides to start saving some money from this upcoming

Jeff decides to start saving some money from this upcoming month onwards. He decides to save only $500 at first, but each month he will increase the amount invested by $100. He will do it for 60 months (including the fir ...

Suppose you make 30 annual investments in a fund that pays

Suppose you make 30 annual investments in a fund that pays 6% compounded annually. If your first deposit is $7,500 and each successive deposit is 6% greater than the preceding deposit, how much will be in the fund immedi ...

Question -under what circumstances is it ethical if ever to

Question :- Under what circumstances is it ethical, if ever, to use consumer information in marketing research? Explain why you consider it ethical or unethical.

What are the differences between four types of economics

What are the differences between four types of economics evaluations and their differences with other two (budget impact analysis (BIA) and cost of illness (COI) studies)?

What type of economic system does norway have explain some

What type of economic system does Norway have? Explain some of the benefits of this system to the country and some of the drawbacks,

Among the who imf and wto which of these governmental

Among the WHO, IMF, and WTO, which of these governmental institutions do you feel has most profoundly shaped healthcare outcomes in low-income countries and why? Please support your reasons with examples and research/doc ...

A real estate developer will build two different types of

A real estate developer will build two different types of apartments in a residential area: one- bedroom apartments and two-bedroom apartments. In addition, the developer will build either a swimming pool or a tennis cou ...

Question what some of the reasons that evolutionary models

Question : What some of the reasons that evolutionary models are considered by many to be the best approach to software development. The response must be typed, single spaced, must be in times new roman font (size 12) an ...

  • 4,153,160 Questions Asked
  • 13,132 Experts
  • 2,558,936 Questions Answered

Ask Experts for help!!

Looking for Assignment Help?

Start excelling in your Courses, Get help with Assignment

Write us your full requirement for evaluation and you will receive response within 20 minutes turnaround time.

Ask Now Help with Problems, Get a Best Answer

Why might a bank avoid the use of interest rate swaps even

Why might a bank avoid the use of interest rate swaps, even when the institution is exposed to significant interest rate

Describe the difference between zero coupon bonds and

Describe the difference between zero coupon bonds and coupon bonds. Under what conditions will a coupon bond sell at a p

Compute the present value of an annuity of 880 per year

Compute the present value of an annuity of $ 880 per year for 16 years, given a discount rate of 6 percent per annum. As

Compute the present value of an 1150 payment made in ten

Compute the present value of an $1,150 payment made in ten years when the discount rate is 12 percent. (Do not round int

Compute the present value of an annuity of 699 per year

Compute the present value of an annuity of $ 699 per year for 19 years, given a discount rate of 6 percent per annum. As