We study

mobility hospital social

Proposal

Introduction / Background

Coronavirus disease 2019 (COVID-19) is rapidly spreading across the globe and is a significant public health threat to humanity infecting millions worldwide. We'll be using machine learning to build models which will be used to forecast casualty rates given novel datasets built from mobility, hospital, and social media data feeds.

Problem Definition

In August, Florida changed how it reported deaths to the CDC by simply reporting “new deaths” they had processed, regardless of what day they actually happened. Since it takes several weeks for data on deaths to be processed, data shown by CDC for the state can show that deaths are falling, even during major outbreaks. (Mendelson, 2021) As such, we aim to build machine learning models to forecast casualty rates by training the model on non-Florida data.Using mean absolute error (MAE), and root mean squared error (RMSE) as our loss function, we hope to measure the efficacy of our models. Should the forecasts prove prescient, we hope that the models can be used by public health officials to quickly mitigate near-term issues affecting their community.

Methods

Data Collection

We have collected and preprocessed data from the following sources:

Source	Granularity	Columns	Rows	Data Size
JHU	State	17	27435	1.8 GB
Google Data	County	15	812066	0.4 GB
Apple Data	County	675	4792	0.5 GB
CDC	State	15	39061	0.8 GB

Data Generation with SIR Modelling

The Susceptible, Infectious, Recovered model is an epidemiology model for modelling the spread of disease within a population. The model consists of three "compartments":

Susceptible

The number of susceptible individuals. These individuals will become infected after coming into contact with an infectious individual.

Infectious

The number of infectious individuals. These individuals are infected and are capable of infecting susceptible individuals.

Recovered

The number of recovered or deceased individuals. These are either immune or deceased individuals who cannot become infectious again.

The system can be described as the following coupled system of ODE $$ \begin{cases} \frac{dS}{dt} = - \beta \frac{IS}{N}, \\ \frac{dI}{dt} = \frac{\beta I S}{N} - \gamma I, \\ \frac{dR}{dt} = \gamma I, \end{cases} $$ where $S$, $I$ and $R$ denotes the number of susceptible, infected and recovered individuals respectively, $N = S + I + R$ is the total population, $\beta$ denotes the infectious rate and $\gamma$ denotes the recovery rate.

We fit the SIR model using the infected and death rate from the CDC and JHU sources to obtain a value of $\beta$ and $\gamma$ from each source for three time intervals: 7, 14 and 30 days. The fitting was performed using scipy.optimize package in python, resulting in 4 columns of data for all 314 days. This data will be used for predicting future infection and casualty numbers.

Data Processing

Cleaning

We first cleaned up the data from various sources into a consistent format. The raw data from the JHU dataset for one of the dates is shown below. In particular, note the missing values for the People_Hospitalized column.

The first 5 rows of the cleaned dataset for Georgia is shown below.

PCA

We next used PCA to perform dimension reduction on our dataset to retain 99% of the variance, resulting in a reduced set of 9-15 features. For Georgia, this resulted in 9 components. The correlation matrix of the features before and after PCA are shown below.

The correlation matrix of the pre-PCA features has large off-diagonal coefficients, indicating strong correlations between features. For example, the confirmed death rates and confirmed death ratios from JHU and CDC are basically identical, with values nearing 1. Similarly, the coefficients of the SIR model coefficients ($\beta$, $\gamma$) from JHU and CDC strongly correlate with each other with values nearing 1.

The correlation matrix of the reduced features is the identity matrix, confirming that PCA has found a set of uncorrelated feature vectors.

To gain more insight into the reduced features, we next visualize how each raw feature contributes to the different PCA components.

Some insights from this are that the SIR coefficients ($\beta$, $\gamma$) have the same direction in the each component. Also, the components for the google-workspaces and google-residences features are strong in opposite directions, matching intuition that staying at home should be equivalent to being away from the workplace.

Results

LSTM-based forecasting

We used PCA and reduced the number of features to 8 for every state. Using this newly generated data, we performed time series based predictions on Covid-19 deaths using LSTM.

Data Preparation

As we saw earlier, we have 50 states and 8 features for each state(from PCA). And every state has data from Jan 1 2021 to August 31 2021. Using this, we prepared the training dataset as follows.

• Use n_past = 30, the number of days in the past to consider for making prediction

• Use n_future=1, number of days in the future we want our prediction to be.

Input to the mode is therefore of dimensions (batch_size, 30, 8) and the predictions will be of the form (batch_size, 1). For example, data from April 1 to April 30 will be combined to a (30,8) vector which will be our input for training and the number of deaths on May 1 will be the label and so on. We scaled the data using MinMaxScaler from sklearn to scale all values between 0 and 1 before PCA and training. We can use the fit_inverse() function on the scaler to scale the results back to the original values.

Hyperparameters

We used the following hyperparameters for LSTM training:

The loss for the LSTM model is shown below.

We chose one of the 50 states and predicted number of deaths for August 1 2021 to August 31 2021 and compared it with the reported number of deaths.

Predictions for some states were accurate (California, Louisiana) while not so much for some states (Colorado).

For most states, we see the predictions made by the LSTM model is either higher/lower than the actual number of deaths reported even though we see that the training and validation loss were less and did not vary significantly

Note that when the model was trained without scaling the data, the loss obtained was significantly off on the order of thousands. While scaling the data did make the loss seem to be less, by scaling the data back to the original vector space we see that the predictions are significantly off, which may indicate that the model is not predicting the deaths accurately.

As a result, we decided to use to decision tree / random forests instead.

Forecasting with Decision Trees / Random Forests

Using the same data, we next tried using the following variations on decision trees / random forests:

The hyperparameters we used are shown below, corresponding to parameters to sklearn.tree.DecisionTreeClassifier:

The following hyperparameters are used for the random forest models (i.e. sklearn.ensemble.BaggingRegressor):

The results for the decision tree / random forest methods are shown below:

Conclusion

We are using four different regressors.

1. The first is the random tree decision tree, which produces a discrete model.
2. The second model, a random forest, was applied as not to overfit the training data in forecasting.
3. The bagged decision tree was bootstrapped, again, to provide some degree of noise in the modeling, and not to overfit the data. Lastly, we bootstrapped a bag of random forests – this randomized some of the features splitting, and randomly sampled the data as to prevent overfitting.
4. Lastly, we applied a weighted ensemble learner, which weighted each regressor based on the RMSE value of the model.

As such, we expect that the decision tree to perform best with short term predictions, and random forest to perform better, the further away the prediction is. We find this to be true for most states. Further, we expect that the bagged random forest to almost always outperform the random forest, as it could account for outliers, and we find this to be true for almost all cases and for all states. The ensemble learner should always perform near the best for all states for all prediction ranges, and we find this to be true for nearly all cases. However, for some findings, we find that the decision tree RMSE to be nearly the same across the board for a specific time predictive time period. Likely, this means that the there isn’t much overfitting. For other states, random forest predictors have high rmse vs decision tree. This could be an outlier in some cases, however, in some states, random forest produced high rmse across multiple predictive time ranges, signifying that likely the decision tree is overfitting on training data. We would expect that the further out predictions are, the worse the performance of the modeling. This is true for almost all states except California and Maine. At first, we suspected this due to be due to Maines sparse data, however, as California also exhibits somewhat convex results on RMSE, we aren’t quite sure what conclusions to draw other than that either the models are not appropriate for these states, or more likely, the input data has errors (given that the modeling is consistent across almost all other states). We did see at the end of our modeling that our predicted values were always under the ground truth data. We later research and discovered this is because decision tree regressors always predict within the range of the values of which it is trained on. We immediately then thought that perhaps this explained the outliers with certain states with random forest, but upon inspection, we did not bootstrap those models, and thus confirm likely the input data for those states likely have errors. We now know that regressors cannot be used to extrapolate data, which does explain our predicted vs forecast results.

Using the features extracted by the PCA, we see that LSTM based Time Series Forecasting proved to be not useful in predicting COVID-19 death rates. From the graphs, we see that for most states the predictions were off by a large margin which made us shift our attention towards other supervised learning models like decision trees and random forests.

References

Mendelson, J. M. (2021, September 28). Florida’s illusion of recovery. Https://Observablehq.Com/. https://observablehq.com/@aloisius/floridas-illusion-of-recovery

Nakkiran, Preetum, et al. “Deep Double Descent: Where Bigger Models and More Data Hurt.” ArXiv.org, 4 Dec. 2019, https://arxiv.org/abs/1912.02292.

Warren, Michael, and Samuel Skillman. “Mobility Changes in Response to COVID-19 - Arxiv.” Mobility Changes in Response to COVID-19, ArXiv, 31 Mar. 2020, https://arxiv.org/pdf/2003.14228.pdf.