Commit fe3193c

#Python for Business Analysts

##Automate Repetitive Tasks

import pandas

housing_2007 = pandas.read_csv("data/Hud_2007.csv")

housing_2005 = pandas.read_csv("data/Hud_2005.csv")

###2: Lists

Now that we have read in both datasets into DataFrame objects, let's add them to a List. A List is type of object (just like a DataFrame, Integer, or String) that contains an ordered group of objects. Just like how a grocery list contains a group of "ingredient" objects to buy, is a List object in Python houses a group of the objects we add to it. Instead of writing code to manipulate each object separately (in our case the objects are DataFrames), we can group a few objects into a List object, write the logic once, and apply it to every object in that List. This saves us a lot of time and energy, and will be important when we deal with much larger datasets with tens, hundreds or even thousands of DataFrames.

Then, we will add a year column for each DataFrame to keep track of which DataFrame is which:

Each row now has a value for year, either 2005 or 2007, identifying which dataset that row originated from. Finally, we will use .append() to first add housing_2005 then housing_2007 to the end of data_frames_list. The List object preserves the order by which the DataFrames were added:

The list now contains these two DataFrames in the order we added them.

data_frames_list.append(housing_2007)

###3: Column Filtering

filtered_housing_2007 = housing_2007[columns]

0 -9 1048 -9 2007

1 69 1048 0 2007

2 45 757 26000 2007

Instead of hard coding the columns we want at the filter level like we did in the last lesson:

we assigned the column names to a list object, called columns:

and passed it into the filter criteria:

Instead of creating two different DataFrame objects (like filtered_housing_2005, filtered_housing_2007, etc), we created an empty list called new_df_list:

and appended each of the filtered_df objects to it:

As you can see, we placed a heavy emphasis on abstracting, or generalizing, our logic so we can detail the logic once, and apply it in many cases. The filter_columns function that we wrote is essentially a piece of software that will filter any list of DataFrame objects into the 4 columns we want. Whether the list of DataFrame objects has 1 DataFrame object or 25, the same function can be applied to get the result we want. Another abstraction we could implement would be to modify the function and specify the columns we want filtered every time by adding it as an input to the function (alongside data_frames_list). This way, instead of always using a specific set of columns within the function, the user can now specify in the input which columns they prefer to filter their DataFrames.

for df in filtered_data_frames_list:

# Print dataframe columns.

###7: Summary

In the following code block:

we use the function str() to convert Integer objects, like year and len(negative_age_count), into String objects. The print function can only print String objects, so we must convert other objects to String objects. While not all objects can be converted to String objects for displaying, most can and we will cover in a later lesson how we can tell.

# Return rows with negative age values.

negative_age_count = df[df['AGE1']<0]

# Print row count.

2005 - 3493 rows

Inside the function, we will first instantiate, or create, an empty list with no elements:

Then, we will iterate through each DataFrame in filtered_data_frames_list, create a temporary DataFrame cleaned_df containing just the positive AGE1 rows for each DataFrame:

And then we will append cleaned_df to cleaned_list for each iteration:

Let's run this function clean_rows on data_frames_list and assign the results to cleaned_data_frames_list.

cleaned_data_frames_list = clean_rows(filtered_data_frames_list)

###11: Verify Cleanup

verification_count = -1

verification_count = verify_cleanup(cleaned_data_frames_list)

###12: Summary