Lab6_Salonis_Paper

docx

School

University of Louisiana, Lafayette *

*We aren’t endorsed by this school

Course

380

Subject

Business

Date

Jan 9, 2024

Type

docx

Pages

Uploaded by emarielaver

Stefanie Salonis BIAM500 – January 2019 Professor Walker Lab 6: Data Mining with a Neural Network Data Mining with a Neural Network Scenario / Summary: Adventure Works Cycles, a fictional bicycle manufacturing and sales company, wants to be able to predict sales from new customers during their first year. Specifically, Adventure Works would like to classify new customers using the following categories.: Category Expected Sales in First Year D Less than $2,000 C $2,000–$2,999 B $3,000–$3,999 A $4,000 or more Adventure Works currently collects a set of demographic data from all customers through a customer survey. Data mining has been performed to determine the feasibility of classifying new customers based on their survey responses and been provided with two data sets extracted from the company's data warehouse: 1. A list of long-term customers with the survey responses and total first-year sales for each ( OldCustomers ) 2. A list of new customers to be classified with their survey responses ( NewCustomers ) Objective – Give findings and recommendations for Adventure Works managers concerning this data mining effort: Include :  An evaluation of the performance of the neural network on this data mining problem / in classifying customers  Recommendations as to how the performance might be improved  Recommendations as to how Adventure Works could use predicted customer classifications to improve business results Findings – How could we proceed if we wanted to try to improve this neural network? Prof. Walker’s Comments : We’ve trained and tested a neural network for customer classification, used the network to predict classifications for some new customers, and evaluated the network’s performance. I’ll leave that mostly for you to think about and write about in your opinion paper, but here are a couple of ideas to get you started. The network may be having trouble learning to recognize A and D customers because there are relatively few of them in the training set, compared to the B and C customers. We might try training the network on a data set that over-samples As and Ds, and under-samples Bs and Cs, compared to the actual distribution. There’s no rule that says the proportions of different case types in your training set have to match reality, and sometimes, there can be benefits to deliberately over-representing or under-

Stefanie Salonis BIAM500 – January 2019 Professor Walker Lab 6: Data Mining with a Neural Network representing certain types of cases. We might also want to think more carefully about which variables should be included in our model. If we are including some irrelevant data that really don’t have anything to do with how much the customer will purchase, this noise might be confusing the network. We could look individually at how each variable is correlated with first-year sales, and eliminate the ones that have weak or no correlation. We might also want to eliminate variables that are highly correlated with other variables; sometimes if two variables have very similar information content, including them both can dilute the impact of either one on the network. You can probably think of some other things to try. Summary Workbook : The Training section tells us the number of training cases, the training time, and the percentage of bad predictions made on the training cases. The most enlightening section is the Testing section , which shows how well the network did on known cases that were not used during the training. This better approximates how the network will perform on new cases in practice. Usually, the testing performance is worse than the training performance. The two numbers to focus on here are the percent of bad predictions . This tells us in how many cases the network's predicted classification did not match the actual classification. It is the percent of bad test cases out of all test cases. We would like this to be a low number. The mean incorrect probability is the average of all the incorrect percent values from the test cases, so it is, again, a measure of confidence. The higher this number is, the less sure the network is of its predictions. Again, we would like this to be a low number.

Stefanie Salonis BIAM500 – January 2019 Professor Walker Lab 6: Data Mining with a Neural Network In evaluating the network, you will probably want to look at the two classification matrices-- one for training and one for testing. Focus on the matrix for testing since it probably gives you a better idea of how the network will do in practice on new previously-unseen cases. The letters down the left side are the actual customer classifications, and the letters across the top are the predicted classifications-- how that network classified these same customers. The number in each cell is the number of cases with that actual and predicted classification. So for example, we can see that, in this run, of all the cases that were actually A customers, the network correctly identified 18 of them as A's. However, it identified 48 of them as B's, 37 of them as C's, and 5 of them as D's. So it correctly identified an A customer only about 17% of the time and it got it wrong about 83% of the time, which is fairly discouraging. Now, it did much better at classifying B customers. Out of all the true B's, the network correctly identified 345 as B's. It classified a true B as A only 10 times; as a C, 96 times; and as a D, 26 times. So on B customers, it is actually right about 72%, 73% of the time and wrong about 27 or 28%. Now, one way to put these results in context is to compare the network's performance to chance. If there were equal numbers of A, B, C, and D customers, then a random guess at the classification for any customer would have a 25% chance of being correct. However, that's not the case. There are many more B's and C's and fewer A's and D's in the customer base. So to see the actual distribution, let's do some quick calculations. In the column next to the classification matrix, enter a sum formula to add the true A's across the predicted A, B, C, and D columns. This gives us the total number of true A's in the sample, and then fill this down to do the same for the true B's, C's, and D's. Then add a sum below to get the total. This should match the total number of test cases. Now, calculate percentages of the grand total in the next column, and remember to make the reference to the grand total absolute in order to be able to fill it down. Format as percentages. This gives us the actual proportions of A's, B's, C's, and D's in the testing sample. These are also the probabilities of picking each classification correctly by chance. So let's head this column "Chance." For comparison, add a column with the neural network's performance. This will be 100% minus the bad percent from the test results. Fill this down, and eliminate the extra decimals. We'll head this column "Neural Net Performance" or just "Net." Now, we can see that only about 10% of our customers are A's, and only about 18% are D's. If we tried to pick A customers by pure chance, we would get it right less than 10% of the time. But the neural network is picking A's correctly about 17% of the time, so it actually may be adding some value here because it's picking at a greater rate than chance. For picking B's, chance would get it right about 42% of the time, but the network gets it right about 72% of the time. So again, the network is definitely adding some value. For picking C's, the network also did slightly better than chance. It actually does about the same-- maybe a little bit better-- than chance for picking D's.

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

Stefanie Salonis BIAM500 – January 2019 Professor Walker Lab 6: Data Mining with a Neural Network Recommended Actions for Management: Based on data, Recommendation is for Management to

Lab6_Salonis_Paper

Related Documents