Assignment

Part-1: Answer the following questions using UNIX.

The dataset (in a file named ChicagoCrimes.csv) that we will use is about crimes in Chicago. It has the following attributes: ID, Case Number, Date, Block, IUCR (Illinois Uniform Crime Reporting), Primary Type, Description, Location Description, Arrest, Domestic, Beat, District, Ward, Community Area, FBI Code, X Coordinate, Y Coordinate, Year, Updated On, Latitude, Longitude, Location.

1. How many primary types are in the data?

2. How many are described as "ATTEMPT THEFT"? How many of those resulted in arrests?

3. Extract all the assault cases (those with primary type "ASSAULT"), sort them by district number and store those records in a separate file.

4. Which are the three most dangerous districts and the three safest in the data?

Make sure to show your UNIX commands for each of these questions. Use any portion of the data used or produced as a way to support your answers (e.g., showing the first or last few records of the data). But you do not need to submit any resulting files from running these commands.

Part-2: Answer the following questions using R.

We will work with a dataset about population estimates. See the attached document Polulation.zip, which has two files -- Population.csv with the dataset and Population.pdf with details of this data and various attributes.

1. Plot a bar chart with top 5 states that have the highest number of international immigration for 2010, 2012, and 2014. (Hint: look for fields INTERNATIONALMIG2010, INTERNATIONALMIG2012, and INTERNATIONALMIG2014)

2. Plot a line graph - population trends over time by (1) region, and (2) division. (Hint: You need to sum the population by region and division. Plotting the line graph with the retrieved dataset may need some data reshaping.)

3. What are the divisions that show the highest increasing rate for population (1) between 2011 and 2012, and (2) between 2013 and 2014?

Copy your R code as well as the outcomes.

Part-3: Answer the following questions using Python.

We will use Online News Popularity dataset for this. Attached is a zip file OnlineNewsPopularity.zip. It contains a csv file and a text file. The former has the data and the latter has the description of the dataset.

1. [Regression] Here, the target variable is ‘shares,' which refers to the number of shares. Do correlation analysis with it and other variables in the dataset to find the most relevant (related) factor (make sure this variable is continuous). Build a regression model using these variables to predict 'shares'. Report the model.

Hint: Try ‘abs_title_sentiment_polarity' for the most relevant factor. You should use that for regression with ‘shares'.

2. [Regression] Make a subset of online news that was published on weekends. Do correlation analysis with 'shares' and other variables in the dataset to find the two most related factors, and build a regression model using these variables to predict 'shares'. Report the model.

3. [Regression] Make a subset of online news that was published on weekdays. Do correlation analysis with 'shares' and other variables in the dataset to find the two most related factors, and build a regression model using these variables to predict 'shares'. Report the model.

4. [Model comparison] Compare the two models from the previous two questions (weekend vs. weekdays) regarding what affects the online news popularity. Report the model (provide your interpretations as you compare those two models).

5. [Clustering] Divide the dataset into two clusters, one having less than 1400 shares, and the other having equal to or greater than 1400 shares. Extract the two features you have from the first question, and show a 2D plot with clusters marked.

Hint: Here's how you can create the required subsets.
newsdata_low = newsdata[newsdata.shares<1400]
newsdata_high = newsdata[newsdata.shares>=1400]

We need two most relevant features to do the plotting. If you had followed the exploration process for Question-1, you should have found those two features. Try ‘title_subjectivity' as the next one (after trying ‘abs_title_sentiment_polarity').

6. [Clustering] Do clustering using two different parameters (k values) for k-means. Show the plots with clusters marked. Provide your interpretations of these two clustering methods in 2-4 sentences comparing them qualitatively.

Hint: select the two columns represented by those two factors we used in Q5.

You can specify a particular point's coordinate like:

newsdata_clustering.iloc[i,0], newsdata_clustering.iloc[i,1]

where i means the row index, 0 refers to 0th column (abs_title_sentiment_polarity), and 1 refers to 1st column (title_subjectivity).

7. [Classification] For classification, take only the two features you've gotten from the first question. Split the dataset into 70% for training and 30% for test using kNN. Show the resulting accuracies with three different values of k.

Hint: To get the classification accuracy, we need the correct label, ‘high' or ‘low' shares, for each instance. Let's get a column and mark if a news item was popular or not.

newsdata_low_extracted['pop'] = 0
newsdata_high_extracted['pop'] = 1

Put this data together in a dataframe.

frames = [newsdata_low_extracted, newsdata_high_extracted]
newsdata_classification = pd.concat(frames)

Now, get our predictors and response variables.

X = newsdata_classification.iloc[:,0:2]
y = newsdata_classification['pop']
Copy your Python code as well as the outcomes.

Attachment:- S-O-S.zip