Scavenge Hunter
In this section, we’ll build a small app that can run on a web browser, preferably on a mobile phone. With Scavenge Hunter, the goal is to collect certain items from a list. We can use parts of the classes list to control the items our user needs to collect and in that case, we’re sure to be able to detect those objects!
Once an object has been detected, we’re going to add a score based on the find and certainty of the model. Since we can’t guarantee that objects are being recognized properly, we should also be able to skip an assignment. Instead of uploading an image, we’re going to use the camera stream!
Setting up the project
We can continue using the prototype we built or create a new project if we’d like. In the case of the latter, the dependencies and store are required, so we’d need to repeat the relevant steps provided in the Setting up the project and Performing and displaying a status check sections.
Let’s see how we can turn the foundation of our prototype into a little game, shall we?
Generic changes
We’re going to start with a configuration file. We need to create this file in the root of the project as config.ts:
export default Object.freeze({ MOTIVATIONAL_QUOTES: [
"Believe in yourself and keep coding!",
"Every Vue project you complete gets you closer to victory!",
"You're on the right track, keep it up!",
"Stay focused and never give up!"
],
DETECTION_ACCURACY_THRESHOLD: 0.70,
SCORE_ACCURACY_MULTIPLIER: 1.10, // input scores are between DETECTION_ACCURACY_THRESHOLD and 1
MAX_ROUNDS: 10,
SCORE_FOUND: 100,
SCORE_SKIP: -150,
})
It can be very helpful to have this sort of configuration files in a central place so that we don’t have to spend time hunting settings down in individual files. Feel free to modify the game configuration values in the config.ts file!
Let’s also open the ./index.html template so that we can update the title tag to the new project’s name – that is, Scavenge Hunter.
We’ll also create two new view files in the ./views folder. It’s okay to just paste some placeholder content here, like so:
<template> <div>NAME OF THE VIEW</div> </template>
We need a view for the finding state, called Find.vue, and one for the end of a game, called End.vue. We’ll add the contents later, in the Building the finish screen and Skipping to the end sections. With the views in place, we can update the ./router/index.ts file with the following contents: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.5-index.ts.
We’re also going to simplify the interface a bit more. In the ./layouts/default folder, delete the AppBar.vue and View.vue files. In the Default.vue file, replace its contents with the following:
<template> <v-app> <v-main> <router-view /> </v-main> </v-app> </template>
Now, we should be able to run the app, but there’s not much new to do at the moment. Let’s add some core features via Pinia stores.
Additional stores
I usually start by designing and setting up the stores since they usually act as a central source of information and methods. First, we’re going to replace the contents of the ./store/app.ts file with contents that are very similar to those from Chapter 6: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.6-app.ts.
It’s a trimmed-down version of the app store we used to build our fitness tracker, but we’ve removed all the unnecessary features.
Since we’re dealing with a predefined list of classes, we’re going to add those to the object.ts store as an additional value:
// ...abbreviatedexport const useObjectStore = defineStore('object', () => {
// ...abbreviated
const loadModel = async () => {
// ...abbreviated
}
loadModel();
// Full list of available classes listed as displayName on the following link:
// https://raw.githubusercontent.com/tensorflow/tfjs-models/master/coco-ssd/src/classes.ts
const objects: string[] = ["person", "backpack", "umbrella", "handbag", "tie", "suitcase", "sports ball", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "orange", "broccoli", "carrot", "chair", "couch", "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "remote", "cell phone", "microwave", "oven", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush"];
return { loadModel, isModelLoading, isModelLoaded, detected, detect, objects }
})
I’ve not added all of the categories and instead selected the classes that we could find in someone’s home. You can change this to what you think is reasonable to have on hand (especially for testing purposes).
Let’s also introduce some game mechanics by adding a ./store/game.ts store file: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.7-game.ts.
This store contains references to the rounds that are being played and which are being skipped (lines 19–23), keeps track of the score (line 23), and helps in selecting a category from the list of objects we’ve defined in the object store. In particular, getNewCategory (lines 28–45) is interesting since it pulls a randomized category from the objects collection while making sure it’s always a unique new category.
As a final step in this section, we’ll replace the contents of the ./App.vue file: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.8-App.vue.
This connects the app store’s capabilities to the interface. Now, we can continue building up our little game!
Starting a new game
We’ll start by creating a button that triggers the conditions for a new game. In the components folder, we’ll create a StartGame.vue component, which is nothing more than a button with some actions on it:
<template> <v-btn
:disabled="!canStart"
@click="newGame"
prepend-icon="mdi-trophy"
append-icon="mdi-trophy"
size="x-large"
color="primary"
><slot>Start game!</slot></v-btn
>
</template>
<script lang="ts" setup>
import { useAppStore } from "@/store/app";
import { useGameStore } from "@/store/game";
import { storeToRefs } from "pinia";
const gameStore = useGameStore();
const appStore = useAppStore();
const { canStart } = storeToRefs(gameStore);
const { reset } = gameStore;
const newGame = () => {
reset();
appStore.navigateToPage("/find");
};
</script>
As you can see, we’re relying on the store to tell the button whether the button should be disabled. We trigger a new game by calling the reset() function of gameStore and calling a navigateToPage function on appStore. Now, we should be able to place this button component on the Home.vue view. Let’s update that view completely with the following contents:
<template> <v-card class="pa-4"> <v-card-title> <h1 class="text-h3 text-md-h2 text-wrap">z Scavenge Hunter</h1> </v-card-title> <v-card-text> <p>Welcome to "Scavenge Hunter"! The game where you find things!</p> </v-card-text> <StatusCheck /> <v-card-actions class="justify-center"> <StartGame /> </v-card-actions> </v-card> </template> <script lang="ts" setup> import StartGame from "@/components/StartGame.vue"; import StatusCheck from "@/components/StatusCheck.vue"; </script>
If you’re running the app now, you’ll notice that it’s impossible to start the game. Since we want to use the user’s camera feed, we need to request access. We’re going to expand the StatusCheck.vue file to also make sure we have access to a camera. We can use a composable from the VueUse library for this. So, from the terminal, let’s install the VueUse package with the following command:
npm i @vueuse/core
With this dependency, we can update the StatusCheck.vue file. The changes to that component are quite extensive, so use the source from https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.9-StatusCheck.vue.
Apart from some additional formatting on our model loading status and some template changes that show the actual status, most changes take place in the script. The usePermission composable returns a reactive property that lets us know if the user has granted access to use the camera. If both the model is loaded and the user has granted camera access, the game can start (lines 61–65). As you can see, we’re using the watch function on multiple values by providing them as arrays (line 61) to the watch function.
In the onMounted hook (lines 67–81), we manually attempt to request a video stream. Once the stream starts, we immediately close it down since we don’t need the stream, just the permission. The permission is persistent throughout our visit.
Building the finish screen
Before we dive into the image streams and object-hunting aspects, we’ll build the final screen. We’ll create a component in the ./components folder to display the result of a game called ScoreCard.vue: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.10-ScoreCard.vue.
In the component, we’re just displaying some of the metrics that were being collected on playthrough. They are all properties that are part of gameStore, so we have easy access to them.
In End.vue, we’ll import the ScoreCard.vue file and make some additions to the template:
<template> <v-card class="pa-4"> <v-card-title> <h1 class="text-h3 text-md-h2 text-wrap">It's over!</h1> </v-card-title> <v-card-text> <p>Let's see how you did!</p> </v-card-text> <ScoreCard /> <v-card-actions class="justify-center"> <StartGame>Play Again?</StartGame> </v-card-actions> </v-card> </template> <script lang="ts" setup> import ScoreCard from "@/components/ScoreCard.vue"; import StartGame from "@/components/StartGame.vue"; </script>
There’s not much going on here apart from the <StartGame /> component, which we have reused to simply trigger a new game. That’s how you use slots! Now, we can work on the middle section!
Skipping to the end
First, let’s make sure we can complete a (very limited) flow by skipping all assignments. We’re going to implement the basic game flow in the ./views/Find.vue file. Let’s take a look at the script tag since we have a lot going on in this file: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.11-Find(script).vue.
At the top of the script tag, we’re loading the properties and methods from the stores (lines 3–15). We use appStore to navigate to different pages and gameStore because that contains information about the progress of the current game.
We have some computed values that help in presenting and formatting data nicely. currentRound (lines 17–19) displays the progress of the game. We use isPlaying (lines 21–23) to determine the boundaries of the rounds versus the maximum set of rounds. Lastly, we have some fun randomized motivational quotes (lines 25–29) that we’ve loaded from our configuration file.
There are two methods in this component. One is to skip (lines 31–39) a round. The skip function tracks the number of rounds skipped (line 32) and modifies the player’s score (lines 33–37). We must make sure the score doesn’t fall below 0. After skipping, we call the newRound method.
The newRound function (lines 41–47) tracks what should happen: either the number of rounds has reached the maximum and we should navigate to the End state, or we should load a new category using the getCategory function from the store. To ensure we get started when we enter this Find state, we will call that newRound function in the onMounted hook.
Next, let’s look at the template of the Find.vue file, where we connect the computed values and methods to a basic interface: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.12-Find(template).vue.
Again, there’s not much special going on here. We’re using the <SkipRound /> component with the @skipped event to make sure we can move forward in rounds, regardless of whether we’ve been able to use object recognition.
Running the app at this stage should give us a result similar to the following:
Figure 9.3 – The basic game flow in place
You should be able to complete the entire flow now by skipping all of the rounds. A game like this makes more sense on a mobile device than a laptop or personal computer, so this would be a good time to make sure we can test the app properly.
Testing on a mobile device
If you’re building an app for a specific use case, it makes a lot of sense to test those cases as early as possible! While we can open the app in mobile views in our browser, it would make a lot of sense to run it on a mobile device as well. The first thing we can do is automatically expose the development server host by updating the dev script in the package.json file:
{ "scripts": {
"dev": "vite --host",
"build": "vue-tsc --noEmit && vite build",
"preview": "vite preview",
"lint": "eslint . --fix --ignore-path .gitignore"
},
"dependencies": {
// ...abbreviated
},
"devDependencies": {
// ...abbreviated
}
}
This change automatically serves the content through your local network, so as long as your mobile device and development server are on the same network, you can access the app via the network address:
Figure 9.4 – Exposing the development server to the network
We’re not there yet, though. The media feed is only accessible over a secure connection. Going with Vite’s recommendation in the official documentation (https://vitejs.dev/config/server-options.html#server-https), we’ll install a plugin for this using the terminal:
npm install --save-dev @vitejs/plugin-basic-ssl
Once the installation is completed, we’ll update the vite.confis.ts file so that it can use the plugin:
// Pluginsimport vue from '@vitejs/plugin-vue'
import vuetify, { transformAssetUrls } from 'vite-plugin-vuetify'
import basicSsl from '@vitejs/plugin-basic-ssl'
// Utilities
import { defineConfig } from 'vite'
import { fileURLToPath, URL } from 'node:url'
// https://vitejs.dev/config/
export default defineConfig({
plugins: [
basicSsl(),
vue({
template: { transformAssetUrls }
}),
// ...abbreviated
],
// ...abbreviated
})
After saving, we can restart the development server. The contents are now served over an HTTPS protocol. It is not using a signed certificate, so you will probably receive a warning from the browser upon first entry. You can now validate each step using your mobile device as well!
With that, we’ve built a basic flow from start to finish and we can test it on a mobile device. The game itself is not very interesting yet though, right? It’s time to add some object recognition to the game!
