![]() ![]() proposed model inversion in 2015 and really opened up the realm of possibilities for extracting data from models. Now being able to take some inputs and de-anonymize them is fun, but what if we could get the model to just tell us all its secrets? That’s where model inversion comes in! Fredrikson et al. Using this “shadow” dataset, we train a simple model to answer the yes or no question: “Was this in the training data?” Then, we can turn our naughty algorithm against Santa’s model - “Dear Santa, was this in your training dataset?” This lets us take real inputs to Santa’s model and find out if the model was trained on that data - effectively letting us de-anonymize the historical nice and naughty lists! Model inversion It doesn’t matter if these in-training and out-of-training data points are nice or naughty - just that we know if they were in the “shadow” training dataset or not. ![]() Then, we can run some additional data through the model for inference and collect the outputs and label it with a “False” value - it was not in the training dataset. We can then take the training data and label the outputs of this model with a “True” value - it was in the training dataset. We can then create a dataset for our “shadow” model - a model that approximates Santa’s nice/naughty system, trained on data that we’ve collected and labeled ourselves. or a tool like PrivacyRaven, an attacker can train a model that figures out whether or not a model has seen an example before.įrom a technical perspective, we know that there is some amount of memorization in models, and so when they make their predictions, they are more likely to be confident on items that they have seen before - in some ways, “memorizing” examples that have already been seen. Membership inference is a class of machine learning attacks that allows a naughty attacker to query a model and ask, in effect, “Was this example in your training data?” Using the techniques of Salem et al. Specifically, the issues of membership inference and model inversion. Unfortunately, these data elves have overlooked some issues in machine learning security. Santa, being a just and equitable person, has already asked his data elves to tackle issues of algorithmic bias. That way, a parent can check whether their child is on the nice or naughty list. Santa has, however, made his model’s API available to anyone who wants it. Thus, Santa has very carefully protected his training data - it’s locked up tight. After all, being on the naughty list can turn one into a social pariah. Santa’s lists have long been a jealously guarded secret. This makes it easy to let the algorithm decide whether they’ll be getting the gifts they’ve asked for or a lump of coal. That’s a lot of children to make a list of, much less check it twice! So like many organizations with big data problems, Santa has turned to machine learning to help him solve the issue and built a classifier using historical naughty and nice lists. According to estimates, there are around 2.2 billion children in the world. Santa’s task of making the nice and naughty list has gotten a lot harder over time. ![]() So, grab an eggnog latte, line up the carols on Spotify, and let’s pick up where we left off. Throughout January, we’ll be publishing Hacky Holidays content (with a few tweaks, of course) to give the new year a festive start. Now that it’s 2022, we’re feeling in need of some holiday cheer, and we hope you’re still in the spirit of the season, too. Editor’s note: We had planned to publish our Hacky Holidays blog series throughout December 2021 – but then Log4Shell happened, and we dropped everything to focus on this major vulnerability that impacted the entire cybersecurity community worldwide. ![]()
0 Comments
Leave a Reply. |