Assignment 1

CSci 39542: Introduction to Data Science¶

Department of Computer Science, Hunter College, City University of New York
Spring 2026

All students must join the course’s gradescope using the provided access code: PK36RW. Students must verify that their Gradescope account has their full name and CUNY email address in the account settings (Account > Edit Account).

Unless otherwise noted, programming assignments are submitted on the course’s Gradescope site and are written in Python. The autograders expect a .py file and do not accept iPython notebooks. Also, to receive full credit, the code should be compatible with Python 3.6 (the default for the Gradescope autograders) and written in good style.

To get full credit for a program, the file must include in the opening comment:

• Your name, as it appears in your Gradescope registration.

• The email you are using for Gradescope.

• A list of any resources you used for the program. Include classmates and tutors that you worked with, along with any websites or tutorials that you used. If you used no resources (other than the class notes and textbooks), then you should include the line: “No resources used.”

For example, for the student, Thomas Hunter, the opening comment of his first program might be:

and then followed by his Python program.

Good style accounts for 5% of the program grade. We are following the standard PEP 8 style guide for Python code. As part of the autograder scripts, your program is run through a static code analyser (aka a “linter”).

• We are using Pylint which reports warnings and errors, including message codes, and then scores the code on a scale from 0 to 10. We multiply your percentage score by 2 (5% of the total) to determine the style grade. For example, if pylint scores your program 9, your style grade is (9/10) * 2 = 1.8.

• See the Pylint website for tutorials, FAQ’s and standard warnings.

• PEP 8 specifies that variable names should be at least 3 characters long, but it is long standing convention that DataFrames are named df. For the autograders, we have included df in the “good-names” that are accepted. This can be done locally with .pylintrc files or using the command-line option pylint --good-names=df.

• Most IDE’s have linting available: for example, see pylint in PyCharm and Linting Python in VSCode.

Program 1: School Counts¶

Due date 26 Febuary 2026

Learning Objective: Refresh students’ knowledge of dictionaries and string functions of core Python, use constant models, and build competency with open source data portals.

Available Libraries: Core Python 3.6+ only.

Data Sources: NYC Open Data:

• 2021 DOE High School Directory

• 2020 DOE High School Directory

• 2019 DOE High School Directory

Sample Datasets:

• 2021_DOE_High_School_Directory_SI.csv (Staten Island schools)

• 2020_DOE_High_School_Directory_late_start.csv (schools with 9am start times in 2020)

NYC OpenData¶

Much of NYC agency data is publicly available at NYC Open Data. We will focus on high schools maintained by the Department of Education.

The raw data for 2021 is available at:

• https://data.cityofnewyork.us/Education/2021-DOE-High-School-Directory/8b6c-7uty

Since it is quite large (1.7MB), let’s filter the data set to just be high schools in Staten Island:

• Click on the OpenData link above. Click on the Actions button. From the dropdown, choose Query data.

• In the Filter region (either at the bottom of the page or on the right-hand side, depending on your browser window size), choose Select a column to filter and choose borocode.

• Change the is one of to is.

• In the search box, put R (for Richmond County or Staten Island).

• Click the Apply button. It will take a few seconds (it’s a large file) but the rows filtered will be all the Staten Island high schools.

• To download the file:

  • Click on the Export button.

  • Under Download, choose CSV.

  • The download will begin automatically (files are usually stored in the Downloads folder).

• Move your CSV file to the directory where you save your programs.

In addition to the Staten Island data set, create several other datasets for testing locally and to practice using the filtering on NYC OpenData.

Most of the data at NYC OpenData is stored in comma-separated-values or CSV format. The first row is usually the column names for the data, separated by commas. Each entry is in its own row, separated by commas. For example, the first 5 rows of entire 2021 dataset are:

For this assignment, we are going to focus on the overview_paragraph column that contains a short description of each school.

Cleaning Data¶

Note: for this assignment, we are not using Pandas. We will recap Pandas in Lecture 2 and use it in the next assignment, Program 2. Instead, we’re going to use the built-in file and string I/O in Python. If you’re rusty on Python, see the chapters in the CSci 127 textbook: DS 100: Section 13.1 (String Methods) and Think CS: Chapter 11 (Files).

Programming assignments are submitted as a single .py file. If you use multiple files or notebooks, convert your program to a single .py file to submit to the autograder. For this first program, we have set up a template (see p1_template.py and also included at the end of this page). Edit the template to include your name, email, and resources and submit to Gradescope to make sure everything works.

Once you have downloaded some test data sets to your device, the next thing to do is format the data to be usable for analysis. Add the following function to your file:

• extract_overviews(file_name): Opens the file_name and from each line of the file, keeps the overview description of the school (the fifth “column”: overview_paragraph). Returns a list of the overviews.

For example, starting with the Staten Island data set (and using the textwrap package to print prettily):

gives the output:

Running the fuction on the late start data set (schools that start 9am or later):

gives the output:

Once you have written your function, test it locally on the small test files. When it works, upload to Gradescope. Given the size of the files that we evaluate your code, you will find it much faster to develop and test the code in your IDE than debugging and testing in Gradescope.

Constant Model¶

The first model that we will use is the constant model which predicts the same (constant) values for all inputs. We saw examples in Lecture 1 and Chapter 4 with modeling restaurant tips and bus lateness. We are going to build two models: one that predicts the length of the overview paragraphs (in number of characters) and a second that predicts the number of sentences (using the number of periods as a proxy for the number of sentences). To do that, we will use some of our datasets to compute values and determine a good constant for each one. Since the datasets can be quite large, instead of storing the numeric values in a list, we will use a dictionary that keeps count of each time a value is seen (if you’re rusty on dictionaries, see Think CS: Chapter 12 or Learning Python 3 from Scratch).

Implement the functions below. In the real world, you would likely combine these into one function, but we are implementing them separately to make partial credit and testing easier:

• count_lengths(overview_list): For each element of the overview_list, the function computes the length (the number of characters in the string). The results are stored in a dictionary of length occurrences where the keys are the lengths seen in overview_list and the values are the number of times each length occurs. Returns the dictionary of length occurrences.

• count_num_sentences(overview_list): For each element of the overview_list, the function computes the number of periods (.) (as a proxy for the number of sentences). The results are stored in a dictionary of occurrences where the keys are the number of periods seen in overview_list and the values are the number of times each occurs. Returns the dictionary of occurrences.

• compute_mean(counts): Computes the mean (average) of counts dictionary weighting each key that occurs by its value (e.g. if the key of 10 has value 8, then the 10 showed up 8 times and adds 10*8 to the computation of the average). Returns the mean.

Continuing our example for the 11 Staten Island high schools:

• Note that two entries have the same length.

Continuing our late start high school example:

gives the output:

We can similarly compute the number of sentences:

gives the output:

For these small examples, the overall lengths of the paragraphs were different, but the number of sentences were much more concentrated.

We can compute the means as well:

gives the output:

Evaluating Our Model¶

The next part of program evaluates how well our constant models do at prediction. We will use a loss function, mean squared error, introduced in Lecture 1 and Section 4.2.

• compute_mse(theta, counts): This function takes two inputs:

  • theta: a numeric value.

  • counts: a dictionary where the keys are numerical and the values are the number of occurrences.

  Computes the Mean Squared Error of the parameter theta and a dictionary, counts. See Section 4.2: Modeling Loss Functions where this function is implemented using numpy and assumes the values are in a list or Series. Note: numpy is not one of the libraries for this assignment and your function should compute MSE without using numpy. Returns the MSE.

Continuing our example of number of sentences in an overview:

gives the output:

The smallest losses are near the mean of 3.8. We can also visualize the lengths (as a histogram in blue) and the loss function in terms of theta (in red) using matplotlib:

This suggests that the best constant model for sentence length is 3.8. Let’s look at the Staten Island data (generated similarly to above) which has mean 4.45:

Looking at the graph of the loss function (red line), it’s minimized closer to 4.5 than 3.8. So, our constant model built on late-starting schools does not do as well for predicting sentences in SI descriptions. We will see in later lectures that the MSE loss function is minimized for the mean value (and that other loss functions achieve their minimum values at different values).

Testing Code¶

Each programming assignment includes functions that test that your code works (a “test suite”). We will first build these in core Python, and in future assignments (Programs 3-6), will introduce standard testing packages.

Your program should include the functions below that test if your functions above perform correctly. Each of these functions takes a function as an argument. You can write them in any order, but we have placed them easiest to hardest below:

• test_compute_mean(mean_fnc=compute_mean):

  This test function takes one input:

  • mean_fnc: a function that takes one input parameter (a dictionary of counts) and returns a numeric value. It has a default value of compute_mean.

  Returns True if the function performs correctly (computes the mean) and False otherwise.

  Hint: we will test your function with three different functions: the correct one, one that returns the mean of the keys (i.e., doesn’t use the counts when computing the mean), and one that always returns 42.

• test_mse(loss_fnc=compute_mse):

  This test function takes one input:

  • mse_fnc: a function that takes in two input parameters (a numeric value and a dictionary of counts) and returns a numeric value. It has a default value of compute_mse.

  Returns True if the function performs correctly (computes mean squared error) and False otherwise.

  Hint: we will test your function with three different functions: the correct one, one that returns the mean squared error of the keys (doesn’t use the counts), and one that always returns 42.

• test_count_lengths(count_fnc=count_lengths):

  This test function takes one input:

  • count_fnc: a function that takes one input parameter (a list of strings) and returns a dictionary. It has a default value of count_lengths.

  Returns True if the function performs correctly (creates a dictionary of counts of occurrences of lengths) and False otherwise.

  Hint: we will test your function with three different functions: the correct one, one that returns the number of strings in the list, and one that always returns a dictionary with the key:value pair 42:42.

Trying first on the correct function:

gives the output:

test_compute_mean(compute_mean) returns True.

Continuing our example:

gives the output:

test_compute_mean( lambda x : 42 ) returns False.

The lambda in Python allows you to write small anonymous functions (see Python Docs 4.8.6 Lambda Expressions for more details).

Program Template¶

For this first program, we have included a template, p1_template.py to use to get started. It includes function stubs and some testing in a (conditionally executed) main function: