Sometimes it's fun to play with code just to see what you can make it do. So I started a small project a few months ago to find a way to mix together machine learning and UI/UX. The project I came up with was simple: update a user's color preferences based on the colors they focus on more as they browse a page. Is this useful? I'm not sure, but it was interesting and I've been able to speak about it at three different conferences!
This is a guide on how I went through this project and some of the things I learned as I got through it. Here's a link to the source code if you're interested: https://github.com/flippedcoder/machine-learning-ui-ux-demos
Background on neural networks
If you don't know anything else about machine learning, you should know a little about neural networks. Neural networks are our loose attempt at replicating a human mind. They're built on algorithms that recognize patterns based on the information we give them. That information could be things like housing prices, stock market values, and user data.
There are a few uses you'll see a lot in web applications like, classification, prediction, and clustering. An example of prediction would be using stock market data you have to pick a range of future values. An example of classification would be deciding if a user's reported message was fraudulent or not. You would take some data and train an algorithm to look at the values you give it and classify the message as legit or not. Lastly, an example of clustering would be checking for troll comments. Normal comments fall within a range of sentiments and statements, but trolls are usually outliers.
Now that you have an idea of what neural networks are, let's talk about how they are built. A neural network is made up of nodes. A node is just like a function. It's a place where a value is calculated. Nodes are modeled after neurons in the brain so they take a set of inputs and give you an output. This is usually the value you are looking for when you decide to use a neural network.
This is about all you need to know to get started with BrainJS. It makes training a neural network incredibly simple. There are a few little things that might be explained as we go along and I'll try to touch on them.
Background on BrainJS
Now let's get into the JavaScript library, BrainJS https://github.com/BrainJS/brain.js#node. This is one of the machine learning libraries out there you can use other than TensorFlowJS. It's easy to use if you know a bit about neural networks and Node and it comes with a lot of great tutorials and documentation. The method I decided to use is the regular neural network training model. Using a recurring neural network is an option, but for this little project a simple neural network was enough.
BrainJS makes it easy to train a model because you can focus more on your dataset and the inputs you want to use instead of trying to learn a new syntax. TensorFlowJS is cool and I've made this same project with it, but I just like BrainJS a little better. Once you go through some of the tutorials, you can just dive in. The most important thing is to actually have something in mind that machine learning would be useful for.
Deciding which features to use
For this project, I was curious if you could train a model to pick out the color on a page that a user prefers and change the elements on the page to a different shade or complement that specific color. The interesting thing is that every time a user refreshes the page, that custom coloring will get reset. That could be a good thing depending on what your application is used for. Making changes like this to the UI might help close a sale or it could encourage someone to fill out a form correctly.
The way I decided to make this project came from a long back and forth between me and a UI designer. I just wanted there to be a way for the user to tell us what they want to see instead of us trying to guess which hues represent the "right message". The data I focused on collecting from the user were the colors of the elements they spent the most time hovering over. Whether or not this was the right approach, I don't know. I just wanted to know what would happen using this metric.
Based on the coordinates of their mouse on the page, I was able to get the RGB values for the element they were over in its current state. My guess was that we subconsciously gravitate towards colors we like so making that subtle change could influence people to move through your application like you want them to. By looking at where they spend the most time on your page, you can take a guess that they are attracted to the colors at some level.
Then you can take that information and adjust the contrast between elements or the overall tone of the website to match that specific user. You can even do it in the browser so you don't have to deal with server-side code. I'm not sure how that works yet, but I read that you can do it in the docs! Now that the features have been defined, the RGB values that users seem to prefer, it's time to start writing some code and training the neural network model.
Training the model to understand user behavior
Just a quick heads up, you won't find all of the code for this project here, but you can check it out in my GitHub repo: https://github.com/flippedcoder/machine-learning-ui-ux-demos There are a few things you need to know before we jump into code. I made this app using create-react-app, I don't have any real data so I just made up the numbers we use, and this was a personal project I thought would be cool to share so the code is… meh. Alright! Let's take a look that the server.js file we'll need.
This is how we create the training data we'll use for our neural network.
const trainingInputData = [
{ green: 0.2, blue: 0.4 },
{ green: 0.4, blue: 0.6 },
{ red: 0.2, green: 0.8, blue: 0.8 },
{ green: 1, blue: 1 },
{ red: 0.8, green: 1, blue: 1 },
{ red: 1, green: 1, blue: 1 },
{ red: 1, green: 0.8, blue: 0.8 },
{ red: 1, green: 0.6, blue: 0.6 },
{ red: 1, green: 0.4, blue: 0.4 },
{ red: 1, green: 0.31, blue: 0.31 },
{ red: 0.8 },
{ red: 0.6, green: 0.2, blue: 0.2 }
];
const trainingOutputData = [
{ dark: 0.8 },
{ neutral: 0.8 },
{ light: 0.7 },
{ light: 0.8 },
{ light: 0.9 },
{ light: 1 },
{ light: 0.8 },
{ neutral: 0.7, light: 0.5 },
{ dark: 0.5, neutral: 0.5 },
{ dark: 0.6, neutral: 0.3 },
{ dark: 0.85 },
{ dark: 0.9 }
];
const trainingData = [];
for (let i = 0; i < trainingInputData.length; i++) {
trainingData.push({
input: trainingInputData[i],
output: trainingOutputData[i]
});
}
The trainingInputData is a sample of values we can expect from the user. So we're looking at the background color of the elements they hover over the most and getting those RGB vales. The trainingOutputData is what we are trying to predict. Everyone has a preference of brightness level and we say that certain RGB values correspond to specific levels of dark, light, and neutral brightness. Based on our input data, we can subtlety adjust the brightness of our page for a user.
Now we can make a new model and use this data to train it as well as get some statistics on how well the model was trained.
const net = new brain.NeuralNetwork({ hiddenLayers: [3] });
const stats = net.train(trainingData);
console.log(stats);
First we made a new neural network with three hidden layers and called it "net". Adding a few hidden layers gives you more accurate results because you are taking advantage of deep learning. Deep learning just means you are using more than one hidden layer to analyze your input. Next we train the neutral network by passing it our training data and we assign it to a variable called "stats". Lastly we'll print all of the statistics like training error and number of iterations in the console.
At this point, the model is ready to run with any inputs you give it. That means it's time to setup some code to get input from the user to feed into the model. The way we'll do this is by getting the input data directly from the user request and create a new training data array for the input. Then we'll run our model with the new training input data to get the values for the new training output data.
Next, we'll update the model with the new training data and run the model on the last input that we get from the user. I chose to do it like that because of the way I'm sending data to the server, which you'll see in more detail the next section. Basically, I'm sending an array of RGB values back to the server in the same shape as the input data. After running the model on the last input, we'll pass that predicted value back to the user and make adjustments to the UI.
app.post('/getUserMouseCoordinates', (req, res) => {
let newTrainingInputData = req.body.userData;
let newTrainingOutputData = [];
let newTrainingData = [];
newTrainingInputData.forEach(input => {
newTrainingOutputData.push(
net.run(
{
red: input.input.red,
green: input.input.green,
blue: input.input.blue
}
)
);
});
for (let i = 0; i < newTrainingInputData.length; i++) {
newTrainingData.push({
input: newTrainingInputData[i],
output: newTrainingOutputData[i]
});
}
net.train(newTrainingData);
let newPredictedValue = net.run(newTrainingInputData[newTrainingInputData.length - 1]);
res.send(JSON.stringify({ predictedValue: newPredictedValue }));
});
Using the model to make updates to the DOM
The user doesn't really care that any of this stuff is happening in the background as long as it doesn't slow them down. So let me show you how I handled getting their input to the server. I wait until there is a decent sized set of user data (100 points) in my input array. Until there are 100 data points, the values get added to the array and we don't send anything to the server. There's no technical reason behind me choosing 100 data points. It's just the number I picked. 🤷♀️
For neural networks, the input and output values fall between 0 - 1. That means if you're working with values larger than that, they need to be scaled to fit within that range. That's the reason I divided each RGB value by 1000. Those values range between 0 - 255 so I wanted to make sure they would all fit within that 0 - 1 range of the neural network. There are tons of better ways to scale values like min-max normalization or the z-score normalization and you should definitely use something like those when you work on a real machine learning project.
Aside from scaling values in the array, when we have 100 data points in the input array we finally get to pass it back to the server. When the input array is passed back and processed, we'll get a response with the predicted values for the light, dark, and neutral composition of the colors the user spent time hovering over. Then we'll update our state variables and call a changeAlpha function which will update the UI for the user.
getUserMouseCoordinates = async (event) => {
if (this.input.length >= 100) {
let userData = { userData: this.input };
let prediction = await fetch('/getUserMouseCoordinates', {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify(userData)
});
const predict = await prediction.json();
this.setState({
light: predict.predictedValue.light,
dark: predict.predictedValue.dark,
neutral: predict.predictedValue.neutral
});
this.changeAlpha();
}
else {
this.input.push(
{ 'input': {
red: event.target.style.backgroundColor.split(',')[0].substr(4) / 1000,
green: event.target.style.backgroundColor.split(',')[1].trim() / 1000,
blue: event.target.style.backgroundColor.split(',')[2].trim().split(')')[0] / 1000
}
}
);
}
}
This is an example of one of my personal projects I had a lot of fun with. I've even gotten to present this project at different tech conferences! The code's not beautiful, the best machine learning practices aren't all in place, and that fine. Project code doesn't have to be perfect when you're trying to figure out if something will work. One day I'll spend the time refactoring and experimenting, but for now, I have other projects I want to finish!
What did you think about this? Is it useful to post and talk about personal projects like this? Would you have liked more detail into the code or more insight on how to come up with random projects like this? I haven't written about my projects like this before so I'm open to feedback!
Hey! You should follow me on Twitter because reasons: https://twitter.com/FlippedCoding