Hands Free Help Using Jaivox Tools

You can use the Jaivox application generator (JAG) to create Android applications that listen continuously to answer questions. This article describes an example of such an application. It also shows the different steps involved in creating similar applications.

By Rajesh John May 12, 2014

For those of us who like to get our hands dirty, the smart phone is a great help. You can have the instructions wherever you are, working on a car, planting a tree, cooking something messy. But it is also a problem because those hands are dirty, and the smart phone is not exactly a dirt-friendly device.

This article describes a way to convert instructions, repair manuals, or cook books into voice assisted, hands free help systems. With such a system, you can get spoken instructions, to questions that you speak.

Example of a voice assisted recipe

We have implemented a recipe for making banana bread. This is really more like a cake, with plenty of eggs, butter and sugar. It contains some ripe bananas making it quite moist.

You can download, build and run the application.

Recipes generally start with some introduction to the item, followed by a list of ingredients and a number of steps. These are easy to read. The problem though is that while mixing the ingredients, you may forget various things that are needed, or the amount needed of each item. This is where the voice assistance becomes useful.

For example, while mixing ingredients you may forget how much flour is needed. If you had an assistant reading the recipe you could ask "how much flour did I need?" and you would get the answer.

The voice assisted recipe fulfills the role of this recipe reading assistant. This mostly involves coming up with a dialog.

A dialog is simply a list of questions and answers. You can make some questions into followup questions, but it is simplest to have independent questions and answers. That way the dialog can handle the user's questions without considering the context of the previous questions and answers.

Designing the dialog

When you are making an unfamiliar recipe, you tend to forget details all the time. The recipe application's dialog considers this. Thus it is meant to handle the kinds of questions people ask.

We have designed an app as an independent app for Android complete with pictures and text. This way you can read the recipe before starting, then have the system respond to various questions you may ask while working on the recipe.

This calls for what we call a "flat dialog". That is one where each question stands independently regardless of context.

The dialog can be specified in various ways. The simplest way is to write it as a question/answer list. Each question here starts at the left margin. The corresponding answer starts below the question, starting with a tab to indicate that it is the answer.

what is this recipe?
  it is for banana bread.
i want to make a cake.
  this recipe is for cake like banana bread.
is that a cake?
  it tastes like a cake.

There are other questions and answers in the dialog, the above lines are just a few of them.

The questions above are all related. A natural dialog would have some variation in the answers given to the same question. Also, many questions may be answered the same way. Thus another way to specify just the three questions above is to put all the questions together, and all the answers together (with a tab in front of each answer.)

what is this recipe?
i want to make a cake.
is that a cake?
  it is for banana bread.
  this recipe is for cake like banana bread.
  it tastes like a cake.

In this form, any of the questions could be answered by any of the answers. The actual answer is selected at random while the app is running.

implementing the app

You can use the Jaivox Application Generator (JAG) version 0.2 to create this as an Android app. To do this first you need a dialog. This can be created within the JAG graphical interface, or it can be created using a text editor and then read into JAG.

After reading or creating the dialog, choose Android as the target (on the right side of the tool) and press the "Generate" button to create an application. For more details please see Creating Android applications.

When the application runs, you will get a screen that has a recipe

This is just like a conventional recipe from a book (images are from Google Images.) You can scroll down to see more steps

You can see a button "Speak" at the bottom of the screen (you do not have to scroll down to see this, it is always visible.) If you tap on this, it will change to "Listening"


Now the app will listen to your questions and answer them, you do not have to tap anything (the app has a timeout in case you do not speak for a long time, this duration can be adjusted in the code.)

When you see the "Listening" button and you ask "What is this recipe?" you will get a response "It is for banana bread." You can ask any question in the dialog. In situations where the Android recognizer does not hear your question well, it may say something like "I did not get that" and you can repeat your question or ask another one.

How to make similar applications

The main work in creating hands free help like the recipe app is to design the dialog. For each situation, you have to think about the questions a user may ask as well as helpful answers.

For example, suppose you want to make an app that helps somebody change a tire. For example there are instructions at Wiki How. This type of information can be converted into a app where you can ask the app (on your Android phone or tablet) various questions while you go about the task.

To create the dialog, think of various questions and answers to those. For example, here is bunch of related questions and answers.

How do I change a tire?
What do I do about a flat tire?
The tire does not seem to have any air.
  You need to jack up the car and replacd the tire.
  You need to take off the bad tire and put in another one.
  The flat tire should be taken off and replaced.

If somebody has never changed a tire, they will have several questions along the way. Designing the right dialog thus involves a lot of thinking about questions.

A good way to start on this task is to consider something like the the Wiki How page above, and to convert each statement given in that web page to several questions.

This approach can be used on several other applications. There are many situations where you need both your hands to do the job. Then it is inconvenient to touch your phone or tablet for instructions, thus voice is the best option. Cooking and auto repair are both situations where your hands are dirty and it is best to get help through voice. Other examples include assembling furniture or toys, repairing wiring or plumbing, cleaning valuable objects or performing medical procedures.