Terminology

Kappa Score

Statistic that measures the agreement between two raters that classify items into mutually exclusive categories and takes into account the probability of random agreement. If two raters are in complete agreement, then the kappa score could be close to 1. If two raters are in complete disagreement, then the kappa score could be close to 0 or even negative if they are agreeing less than the probability of random agreement.

Confusion Matrix

A table that is used to display the performance of a classification model on a test data set as compared to the ground truth. Each square has a unique row and column combination. The square in row 1 and column 2 says that the ground truth classified those test data points as category 1 but the classification model classified those same test data points as category 2. All of the squares along the diagonal of the table display the number of test cases when the ground truth and classification model are in agreement. All squares off the diagonal represent any disagreement between the ground truth and classification model; the further away from the diagonal, the larger the disagreement. If there is complete agreement between the ground truth and classification model, then all of the off diagonal squares will be zero. If there is complete disagreement between the ground truth and classification model, then all of the diagonal squares will be zero.

Training Dataset Size

The size of the training dataset can affect the accuracy of the model. In general, the model will be more accurate as the training dataset size increases.

Data Set

The dataset that contains the prepared images that you train the model with.

If no data sets are listed, you will need to prepare them first. keyboard_backspace Go To Prepare

Augmentation

Apply additional transformations, such as rotations, flips, or stretching, to images input to the network.

• None: Do not transform images.
• Random Flips/Rotations: Add random flips and rotations to the scans before the model sees them.

Architectures

Structure of the neural network.

• ResNet: Type of architecture developed in 2015 for the ImageNet Large Scale Visual Recognition Challenge that introduced skipped connections between layers and won first place in image classification, detection, and localization.
• VGGNet: Type of architecture developed in 2014 for the ImageNet Large Scale Visual Recognition Challenge that won first place on image localization.
• InceptionNet: Type of architecture whose innovation was to attempt to process information at different spatial scales. It also used new methods to make neural networks "deeper" (more complex) while still learning broad, generalizable concepts about its chosen task.
• DenseNet: Type of architecture whose innovation in neural networks allowed them to maintain high levels of complexity and classification ability, while greatly reducing the computational time and memory required to classify a single case.

Architecture Videos

Sampling Methods

Technique for selecting the studies that the model sees at each training step.

• Equal Class Ratios: Sample the studies so that each class is represented equally – 25% almost entirely fat, 25% scattered fibroglandular densities etc.
• Random Sampling: Sample the studies randomly, in the distribution they are found in the original dataset.

Loss Functions

How the model knows what to learn

• Mean Absolute Error
  • Treat all successes differently depending on confidence, treat all failures differently depending on confidence.
  • If the model got the right category and was very confident in the choice, then it was a large success. If the model got the right category and was not confident in the choice, then it was a small success. If the model got the wrong category and was not confident in the choice, then it was a small failure. If the model got the wrong category and was very confident in the choice, then it was a large failure.
  • Assume that the distance between the categories is the same.
• Categorical Cross-Entropy
  • Treat all successes differently depending on confidence, treat all failures equally.
  • If the model got the right category and was very confident in the choice, then it was a large success. If the model got the right category and was not confident in the choice, then it was a small success. If the model got the wrong category, then it was a failure and all failures are treated the same.
• Mean-Squared Error
  • Treat all successes differently depending on confidence, treat all failures differently depending on confidence.
  • If the model got the right category and was very confident in the choice, then it was a large success. If the model got the right category and was not confident in the choice, then it was a small success. If the model got the wrong category and was not confident in the choice, then it was a small failure. If the model got the wrong category and was very confident in the choice, then it was a large failure.
  • Heavily penalize large failures and assume that the distance between the categories is the same.

Loss function Videos

Pretraining

Whether or not the model has been trained on a different task before training on a new task

• Random Initialization: Randomly start the model, it has not been pretrained on another task.
• Pretrained Weights (ImageNet): Start with a pretrained model that was previously trained to classify natural images, such as photos of cars, dogs, and buildings.

Batch Size

The number of training examples the model learns from during a single step. If the batch size is too small, it can lead to chaotic training because the model doesn't see enough examples at once. If the batch size is too large, the training may fail due to memory constraints of the server. If the training keeps failing, try the smallest batch size possible and increase until the memory limit is exceeded.

Initial Learning Rate

How much the model should change while learning from the training data. A learning rate that is too small can lead to long training times and could cause the model to get 'stuck'. A learning rate that is too high can lead to chaotic training because the model is changing too much.

Improvement Interval

Number of epochs the model will wait for improvement before early stopping. If early stopping is True, then the model will continue to train for a minimum improvement interval. If no improvement is made, then the model will stop training early. If there is improvement, the interval restarts. If early stopping is False, the model will not stop training early.

Early Stopping

If there is early stopping, the model will stop learning once its performance is no longer improving.

Performance Metric

The metric being used to measure the performance of the algorithm.

Kappa Score

Statistic that measures the agreement between two raters that classify items into mutually exclusive categories and takes into account the probability of random agreement. If two raters are in complete agreement, then the kappa score could be close to 1. If two raters are in complete disagreement, then the kappa score could be close to 0 or even negative if they are agreeing less than the probability of random agreement.

AUC Score

(Area under the curve of the Receiver operating characteristic) Performance measurement for classification problems that measures how well a model separates categories. An AUC score close to 1 means that the model is good at separating categories; it classifies 0s as 0s and 1s as 1s. An AUC score close to 0 means that the model is separating into opposite categories; it classifies 0s as 1s and 1s as 0s. An AUC score close to 0.5 means that the model is no better at separating categories than random chance.

Confusion Matrix

A table that is used to display the performance of a classification model on a test data set as compared to the ground truth. Each square has a unique row and column combination. The square in row 1 and column 2 says that the ground truth classified those test data points as category 1 but the classification model classified those same test data points as category 2. All of the squares along the diagonal of the table display the number of test cases when the ground truth and classification model are in agreement. All squares off the diagonal represent any disagreement between the ground truth and classification model; the further away from the diagonal, the larger the disagreement. If there is complete agreement between the ground truth and classification model, then all of the off diagonal squares will be zero. If there is complete disagreement between the ground truth and classification model, then all of the diagonal squares will be zero.

Training Dataset Size

The size of the training dataset can affect the accuracy of the model. In general, the model will be more accurate as the training dataset size increases.

Augmentation

Apply additional transformations, such as rotations, flips, or stretching, to images input to the network.

• None: Do not transform images.
• Random Flips/Rotations: Add random flips and rotations to the scans before the model sees them.

Prepared Dataset

.......

Architectures

Structure of the neural network.

• ResNet: Type of architecture developed in 2015 for the ImageNet Large Scale Visual Recognition Challenge that introduced skipped connections between layers and won first place in image classification, detection, and localization.
• VGGNet: Type of architecture developed in 2014 for the ImageNet Large Scale Visual Recognition Challenge that won first place on image localization.
• InceptionNet: Type of architecture whose innovation was to attempt to process information at different spatial scales. It also used new methods to make neural networks "deeper" (more complex) while still learning broad, generalizable concepts about its chosen task.
• DenseNet: Type of architecture whose innovation in neural networks allowed them to maintain high levels of complexity and classification ability, while greatly reducing the computational time and memory required to classify a single case.

Architecture Videos

Sampling Methods

The ratio of each class that the model sees at each training step

• Equal Class Ratios: Sample the studies so that each class is represented equally – 25% almost entirely fat, 25% scattered fibroglandular densities etc.
• Random Sampling: Sample the studies randomly, in the distribution they are found in the original dataset.

Loss Functions

How the model knows what to learn

• Mean Absolute Error
  • Treat all successes differently depending on confidence, treat all failures differently depending on confidence.
  • If the model got the right category and was very confident in the choice, then it was a large success. If the model got the right category and was not confident in the choice, then it was a small success. If the model got the wrong category and was not confident in the choice, then it was a small failure. If the model got the wrong category and was very confident in the choice, then it was a large failure.
  • Assume that the distance between the categories is the same.
• Categorical Cross-Entropy
  • Treat all successes differently depending on confidence, treat all failures equally.
  • If the model got the right category and was very confident in the choice, then it was a large success. If the model got the right category and was not confident in the choice, then it was a small success. If the model got the wrong category, then it was a failure and all failures are treated the same.
• Mean-Squared Error
  • Treat all successes differently depending on confidence, treat all failures differently depending on confidence.
  • If the model got the right category and was very confident in the choice, then it was a large success. If the model got the right category and was not confident in the choice, then it was a small success. If the model got the wrong category and was not confident in the choice, then it was a small failure. If the model got the wrong category and was very confident in the choice, then it was a large failure.
  • Heavily penalize large failures and assume that the distance between the categories is the same.

Loss function Videos

Pretraining

Whether or not the model has been trained on a different task before training on a new task

• Random Initialization: Randomly start the model, it has not been pretrained on another task.
• Pretrained Weights (ImageNet): Start with a pretrained model that was previously trained to classify natural images, such as photos of cars, dogs, and buildings.

Early Stopping

If there is early stopping, the model will stop learning once its performance is no longer improving.

Epochs

The number of times the model learns from the training dataset.

Kappa Score

Statistic that measures the agreement between two raters that classify items into mutually exclusive categories and takes into account the probability of random agreement. If two raters are in complete agreement, then the kappa score could be close to 1. If two raters are in complete disagreement, then the kappa score could be close to 0 or even negative if they are agreeing less than the probability of random agreement.

AUC Score

(Area under the curve of the Receiver operating characteristic) Performance measurement for classification problems that measures how well a model separates categories. An AUC score close to 1 means that the model is good at separating categories; it classifies 0s as 0s and 1s as 1s. An AUC score close to 0 means that the model is separating into opposite categories; it classifies 0s as 1s and 1s as 0s. An AUC score close to 0.5 means that the model is no better at separating categories than random chance.

Confusion Matrix

A table that is used to display the performance of a classification model on a test data set as compared to the ground truth. Each square has a unique row and column combination. The square in row 1 and column 2 says that the ground truth classified those test data points as category 1 but the classification model classified those same test data points as category 2. All of the squares along the diagonal of the table display the number of test cases when the ground truth and classification model are in agreement. All squares off the diagonal represent any disagreement between the ground truth and classification model; the further away from the diagonal, the larger the disagreement. If there is complete agreement between the ground truth and classification model, then all of the off diagonal squares will be zero. If there is complete disagreement between the ground truth and classification model, then all of the diagonal squares will be zero.

Instructions

• Prepare the data.
  • Select the Usecase
  • Select the training data set.
  • Choose Prep for Annotation
  • Choose Prep for Model Creation
  • Click Prepare Data to process the dataset for Annotation and Model Creation

Use Cases

Select the use case for which you would like to prepare the data for.

Data Set

Select the data set that contains the images that you would like to prepate.

If no data sets are listed, you will need to create them first in Data Manager. keyboard_backspace Go To Data Manager

Prep for Annotation

Select to convert the Dicom images on the dataset into png images for annotation purposes.

Prep for Model Creation

Select to convert the Dicom images in the dataset into Numpy objects for model creation.