Repository navigation
add stream importance to mmn(), and fix three defects large fits exposed - #127
Merged
Merged
Conversation
summary() on an mmn() model repeated print(). It now reports what each data stream contributed: one stream is shuffled across the batch while the others are left intact, and the increase in loss is that stream's contribution. This is the interpretive tool a multi-modal model admits, since per-feature effects are not comparable across modalities of different kinds. The standard deviation across permutations is reported beside the estimate, because a stream that contributes little can come out negative by chance; on simulated data where the images carry the signal and the covariates almost none, the two come out as 0.134 +/- 0.005 and 0.031 +/- 0.014. plot() works on a fitted MMN. plot_type = "arch" draws the model as the graph it is -- one row per data stream with its submodel, the width each stream contributes to the concatenated representation, and the fusion network on top -- instead of drawing each convolutional submodel's layers in turn and erroring out when there is no convolutional submodel at all. mmn() gains weights, which dnn() and cnn() already had. The argument alone was not enough: has_weights was hardcoded FALSE in training_properties, so train_model() would have handed the weights to the network as another input rather than to the likelihood. Zeroing half the weights now halves the loss, 0.2525 -> 0.1305. predict() on an MMN checks newdata against the fitted model. A wrong image size surfaced as "mat1 and mat2 shapes cannot be multiplied" from several frames down, and streams of differing length were silently truncated to the shortest. The three defects below were invisible in ordinary use and only appeared when the package was pointed at a real analysis. A fitted model carried its training data. get_loss() takes five arguments and forced none of them; the loss module's methods close over that frame, so an unevaluated promise kept the caller's frame alive, and with it X. saveRDS() then wrote the whole training set into the model: 3.3 MB of images produced a 4.8 MB file, and the cross-validation models of a 15k-image analysis reached 14 GB each, 137 GB for ten models whose weights are 9.4 MB. object.size() does not follow a closure's environment, which is why every component looked small. With the arguments forced, the saved object no longer scales with the data -- X grown sixteenfold leaves it at 0.56 MB. The validation loop built an autograd graph it never used. $train(FALSE) switches dropout and batch normalization but leaves autograd on, so monitoring a validation split cost as much memory as training on it. Fine-tuning a pretrained backbone with a validation split ran out of memory on a 24 GB card at 23.3 GB in a single process; under with_no_grad() the same fit uses 16 to 19 GB. The losses are unchanged. predict() on an MMN returned a matrix without row names, where predict() on a cnn() and a dnn() set them. format_input_data() converts each stream to a tensor, which carries no dimnames, so the names have to be taken before that. Without them a lookup such as metadata[rownames(pred), ] returns a zero-length vector and a downstream accuracy evaluates to NA instead of raising an error. tests/testthat/test-regressions.R pins all three: the saved model must not grow with the data, a validation split must still produce a usable loss curve, and predict() must carry the names through and must not invent any when the input has none. The MMN section of the vignette was not runnable. It passed a bare response name without dataList, gave predict() an unnamed list, and claimed cito applies no inverse link. Every chunk was eval=FALSE, which is why none of it had been caught. It now carries a worked example that runs at build time and ends in summary() and plot().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary() on an mmn() model repeated print(). It now reports what each data stream contributed: one stream is shuffled across the batch while the others are left intact, and the increase in loss is that stream's contribution. This is the interpretive tool a multi-modal model admits, since per-feature effects are not comparable across modalities of different kinds. The standard deviation across permutations is reported beside the estimate, because a stream that contributes little can come out negative by chance; on simulated data where the images carry the signal and the covariates almost none, the two come out as 0.134 +/- 0.005 and 0.031 +/- 0.014.
plot() works on a fitted MMN. plot_type = "arch" draws the model as the graph it is -- one row per data stream with its submodel, the width each stream contributes to the concatenated representation, and the fusion network on top -- instead of drawing each convolutional submodel's layers in turn and erroring out when there is no convolutional submodel at all.
mmn() gains weights, which dnn() and cnn() already had. The argument alone was not enough: has_weights was hardcoded FALSE in training_properties, so train_model() would have handed the weights to the network as another input rather than to the likelihood. Zeroing half the weights now halves the loss, 0.2525 -> 0.1305.
predict() on an MMN checks newdata against the fitted model. A wrong image size surfaced as "mat1 and mat2 shapes cannot be multiplied" from several frames down, and streams of differing length were silently truncated to the shortest.
The three defects below were invisible in ordinary use and only appeared when the package was pointed at a real analysis.
A fitted model carried its training data. get_loss() takes five arguments and forced none of them; the loss module's methods close over that frame, so an unevaluated promise kept the caller's frame alive, and with it X. saveRDS() then wrote the whole training set into the model: 3.3 MB of images produced a 4.8 MB file, and the cross-validation models of a 15k-image analysis reached 14 GB each, 137 GB for ten models whose weights are 9.4 MB. object.size() does not follow a closure's environment, which is why every component looked small. With the arguments forced, the saved object no longer scales with the data -- X grown sixteenfold leaves it at 0.56 MB.
The validation loop built an autograd graph it never used. $train(FALSE) switches dropout and batch normalization but leaves autograd on, so monitoring a validation split cost as much memory as training on it. Fine-tuning a pretrained backbone with a validation split ran out of memory on a 24 GB card at 23.3 GB in a single process; under with_no_grad() the same fit uses 16 to 19 GB. The losses are unchanged.
predict() on an MMN returned a matrix without row names, where predict() on a cnn() and a dnn() set them. format_input_data() converts each stream to a tensor, which carries no dimnames, so the names have to be taken before that. Without them a lookup such as metadata[rownames(pred), ] returns a zero-length vector and a downstream accuracy evaluates to NA instead of raising an error.
tests/testthat/test-regressions.R pins all three: the saved model must not grow with the data, a validation split must still produce a usable loss curve, and predict() must carry the names through and must not invent any when the input has none.
The MMN section of the vignette was not runnable. It passed a bare response name without dataList, gave predict() an unnamed list, and claimed cito applies no inverse link. Every chunk was eval=FALSE, which is why none of it had been caught. It now carries a worked example that runs at build time and ends in summary() and plot().