TDM 10100: Project 5 Question 5 examples

Now consider the CMS Diabetes Prescriber data:

import pandas as pd
myDF = pd.read_csv("/anvil/projects/tdm/data/CMS/diabetes_prescriber_data.csv")

We can consider the head of the data:

myDF.head()

The Prscrbr_State_Abrvtn contains the state:

myDF['Prscrbr_State_Abrvtn'].value_counts()

but Python does not have a built-in list of the 50 states. We can make such a list:

state_abb = [
    "AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE", "FL", "GA",
    "HI", "ID", "IL", "IN", "IA", "KS", "KY", "LA", "ME", "MD",
    "MA", "MI", "MN", "MS", "MO", "MT", "NE", "NV", "NH", "NJ",
    "NM", "NY", "NC", "ND", "OH", "OK", "OR", "PA", "RI", "SC",
    "SD", "TN", "TX", "UT", "VT", "VA", "WA", "WV", "WI", "WY"
]

These are rows of the data in which the Prscrbr_State_Abrvtn is found in state_abb:

myDF[myDF['Prscrbr_State_Abrvtn'].isin(state_abb)].head()

We can use a tilde (which is a negation in Python) to find rows of the data that are not in this list of states:

myDF[~myDF['Prscrbr_State_Abrvtn'].isin(state_abb)].head()

and now we can check how many times each of those values occurs:

myDF[~myDF['Prscrbr_State_Abrvtn'].isin(state_abb)]['Prscrbr_State_Abrvtn'].value_counts()

and we see that there are, for instance, 383 rows from Puerto Rico, and 158 rows from DC, etc.