Description

A real-time landscape view generator with the sketch painted by user’s hand movement as the input.

Summary

Machine learning models: Posenet/Spade-coco

JavaScript Library: p5.js/ml5.js/tone.js

Other sources: Runway

Composition

Posenet collects the user’s wrists real-time position data

Send the data to

P5.js where it’s translated to the stroke

Send the picture to

Spade-coco where it’s transformed to landscape view

Send the result back to

P5.js

Difficulty

The data from Posenet is not stable.

In-sketch function

The user can pick 5 colors of the stroke which stand for: Cloud/Grass/Mountain/Sea/Tree with left hand staying on the palette at left

Only when the left hand is on the palette the right hand can paint

By choosing different colors, the user can change the BGM

By moving the right hand, the user can change the pitch/volume

By crossing two wrists the user can refresh and download the sketch

Expectation&Future

As its name, I want to offer a fake experience of making a mirage out of window.

The purpose is to have a look of what people will do with the project

To add guideline in sketch

To make the music downloadable