Talon

From Giki

Contents

FEATURE TO TEST

  • Add actions for some Talon menu items, see actions.list('menu').

mishearings

what i said what talon understood
series serious
urge each
or for/four
press plus

alternatives for actions/commands/symbols

alternatives for when the "defaults" (knausj-repo) are not well recognized merged with some from tavis rudds alphabet (https://gist.github.com/tavisrudd/5361092)

alphabet alternative
air axel
air aim
air arch
bat bob
bat batch
bat bill
cap carl
drum dave
each birch
each eric
fine fox
fine faint
gust gary
harp ham
harp hank
harp hoop
sit ivan
sit ice
sit ida
sit ish
sit ivy
jury jack
jury jane
jury jail
jury joe
jury judge
krunch kirk
krunch keel
look luka
look little
made murphy
made mary
made met
made mike
near nate
near noy
odd oz
odd oink
odd orange
odd osh
pit prince
pit palm
pit pink
pit pom
quench quebec
quench queen
red robin
red ree
sun smith
sun soi
trap timy
trap tay
trap tea
urge usher
vest van
vest vince
whale will
whale wet
whale whip
whale wish
plex xin
yank yugi
yank yep
yank young
zip zoko
zip zone
zip zoo
zip zulu
actions/commands/symbols alternative
redo yup
tilde (~) wiggle
undo nope
undo fuck
pah space
void space
clap enter

Talon with remote Dragon

source: https://news.ycombinator.com/item?id=24850417

lunixbochs 4 months ago [–]

Talon can also use Dragon remotely from a Windows machine or VM, kind of like aenea but more tightly integrated (the host is driving instead of the guest).

And I do have some users voluntarily switching from Mac Dragon to the improved speech engine in the Talon beta. Mac Dragon has been kind of buggy for the last few years so you're not missing much.

	
	
ianhorn 4 months ago [–]

Any chance you have pointers on how to set that up? You'd probably laugh/cry to see my setup right now, with my Windows desktop on the left monitors and my MacBook on the right monitors because I need both... purely because Dragon is only sold on Windows since this started been an issue for me. A more tightly coupled super-aenea sounds pretty fantastic.

	
	
lunixbochs 4 months ago [–]

Sure: First run Talon on both sides. Then go to your talon home directory (click talon icon in tray -> scripting -> open ~/talon). There's a draconity.toml file there.

On the Dragon side, you need to uncomment and update the `[[socket]]` section to listen on an accessible IP.

On the client side (Mac in your case), uncomment / update the `[[remote]]` section to point at your other machine.

You also need to make sure both configs have the same secret value.

From there, restart Dragon and look in the Talon log on the Mac side for a line like "activating speech engine: Dragon".

To prevent command conflicts, I recommend setting Dragon to "command mode" (if you have DPI), and only adding scripts to Talon on the Mac side.

If it doesn't work, you can uncomment the `logfile` line in draconity.toml on the Dragon side, restart Dragon, and look in the log to see what's going on. 

linux gnome remarks

also if you’re on latest gnome you should install appindicator support, they removed that for some reason

linux notifications

Put this in notify.py
from talon import Module, Context, app, speech_system, actions
ctx = Context()
mod = Module()
def on_phrase(j):
    phrase = getattr(j["parsed"], "_unmapped", j["phrase"])
    phrase = " ".join(word.split("\\")[0] for word in phrase)
    app.notify(phrase)
speech_system.register('post:phrase', on_phrase)
# above enables notifications. Below supports toggling them.
do_notify = True
@mod.action_class
class Actions:
    def notify_toggle():
        """Toggle OS notifications"""
        global do_notify
        if do_notify:
            speech_system.unregister('post:phrase', on_phrase)
        else:
            speech_system.register('post:phrase', on_phrase)
        do_notify = not do_notify
3:53
And in a .talon file: talon notify: user.notify_toggle().

linux pull all repos in ~/.talon/user

source: https://talonvoice.slack.com/archives/CKYAFAADA/p1651807016464379

find ~/.talon/user -name .git -print -execdir git pull --ff-only \;

load things on talon startup

from talon import app, speech_system
def on_launch():
    speech_system.mimic("command history")
app.register('launch', on_launch)

alternative paste method

in case anyone else is finding the actions.user.paste() problematic (too slow) on linux, you can use this replacement code in code/edit.py in the interim, which seems to be work fine based on testing (not sure if someone posted another workaround already):

@mod.action_class
class Actions:
    def paste(text: str):
        """Pastes text and preserves clipboard"""
        #with clip.revert():
        #    clip.set_text(text)
        #    actions.edit.paste()
        #    # sleep here so that clip.revert doesn't revert the clipboard too soon
        #    actions.sleep("150ms")
        # TODO: qt clipboard is bugged, do it manually
        old = subprocess.check_output(['xclip', '-o', '-sel', 'clip'])
        clip.set_text(text)
        actions.edit.paste()
        actions.sleep("150ms")
        clip.set_text(old.decode("utf-8")) 

note you may have to install xclip depending on your distro (edited)

start talon in sleep mode

from talon import actions, app
def on_ready():
    actions.speech.disable()
app.register('launch', on_ready)

change to webengine (mac specific)

from talon import speech_system, Context
from talon.engines.w2l import WebW2lEngine, W2lEngine
from talon.engines.webspeech import WebSpeechEngine
w2l = W2lEngine(model='en_US-sconv-beta6', debug=True)
speech_system.add_engine(w2l)
webspeech = WebSpeechEngine()
speech_system.add_engine(webspeech)
# set the default engine
ctx = Context()
ctx.settings = {
    'speech.engine': 'wav2letter',
}

from talon.mac import applescript
import subprocess
if applescript.run('''
on webSpeechRunning()
    if application id "com.google.Chrome" is not running then return false
    tell application id "com.google.Chrome" to get (every window's every tab's URL as string) contains "http://localhost:7419/"
end webSpeechRunning

webSpeechRunning()
''') == 'false':
    subprocess.run(['/usr/local/bin/launch', '-bi', 'com.google.Chrome', 'http://localhost:7419'])
    subprocess.run(['/usr/local/bin/appswitch', '-hi', 'com.google.Chrome'])

draw image

from talon.canvas import Canvas
from talon.skia import Image
im1 = Image.from_file("image1.png") # current directory is ~/.talon
def draw(canvas):
    canvas.draw_image(im1, canvas.x, canvas.y)
canvas = Canvas(0, 0, 200, 200)
canvas.register('draw', draw)
canvas.freeze()

react on hiss

from talon import noise, ctrl, cron
import time
​
active_hiss = {"start": None, "cron": None}
HISS_MINIMUM_THRESHOLD = 0.25
​
def hiss_over_threshold():
    if not active_hiss["start"]:
        return False
​
    delta = time.time() - active_hiss["start"]
    trigger = delta > HISS_MINIMUM_THRESHOLD
    if delta:
        print("hisssing complete %0.2fs (trigger = %s)" % (delta, trigger))
​
    return trigger
​
def stop_hiss():
    if hiss_over_threshold():
        ctrl.mouse_click(0)
​
    if active_hiss["cron"]:
        cron.cancel(active_hiss["cron"])
        active_hiss["cron"] = None
​
    active_hiss["start"] = None
​
def start_hiss():
    if not active_hiss["start"]:
        active_hiss["start"] = time.time()
​
    active_hiss["cron"] = cron.interval("16ms", check_hiss)
​
def on_hiss(noise):
    if noise:
        start_hiss()
    else:
        stop_hiss()
​
def check_hiss():
    delta = float(time.time() - active_hiss["start"])
    should_click = delta > HISS_MINIMUM_THRESHOLD
​
    print("hisssing check: %0.2f s"% delta)
​
    if should_click:
        stop_hiss()
​
noise.register('hiss', on_hiss)

vs code syntax highlight extension

source: https://marketplace.visualstudio.com/items?itemName=mrob95.vscode-talonscript

add german dictation mode (using `vosk`)

source: https://github.com/mqnc/talon_german

script: print all talon commands

source: https://gist.github.com/tararoys/c538b7ae8e1f21db9a794c2c0f5becf4

script: reminder

source: https://gist.github.com/tararoys/d138b36731262fb8ae452814c13b9d4f

script: full mouse grid

https://github.com/tararoys/modified_full_mouse_grid

script: pokey cursorless

https://github.com/pokey/pokey_talon/tree/master/apps/vscode/cursorless

script: text selection

https://github.com/splondike/talon_telector

script: talon hud

https://github.com/chaosparrot/talon_hud

setup WebSpeech for Dictation

file: user/engines.py

webspeech = WebSpeechEngine()
speech_system.add_engine(webspeech)

# set the default engine
ctx = Context()
ctx.settings = {
    'speech.engine': 'wav2letter',
}

file: user/webspeech.talon

mode: dictation
-
settings():
    speech.engine = 'webspeech'
    speech.language = 'en_US'

While Talon is running, open http://localhost:7419 in Chrome. You'll probably also need to allow microphone access the first time you enable it.


german webspeech mode

https://github.com/knausj85/knausj_talon/compare/master...jpaddison3:german-web-speech

practice talon

https://chaosparrot.github.io/talon_practice/

telector fix suggestion

https://github.com/splondike/talon_telector/blob/94381bf9a9d6bb7f513275a885fcdb0ee66b7917/talon_interface.py#L374

6:12
try replacing 391 - 404 with this:
src = (rect1.x, rect1.y + rect1.height / 2)
dst = (rect2.x + rect2.width+2, rect2.y + rect2.height / 2)
actions.mouse_move(*src)
actions.mouse_click()
time.sleep(0.1)
actions.mouse_move(*dst)
actions.key('shift:down')
actions.mouse_click()
actions.key('shift:up')
time.sleep(0.1)

knausj formatters actions

settings():
    user.code_private_function_formatter = "SNAKE_CASE"
    user.code_protected_function_formatter = "SNAKE_CASE"
    user.code_public_function_formatter = "SNAKE_CASE"
    user.code_private_variable_formatter = "SNAKE_CASE"
    user.code_protected_variable_formatter = "SNAKE_CASE"
    user.code_public_variable_formatter = "SNAKE_CASE"

code block example with optional formatters

if [<user.formatters>] <user.text> then:
    formatters = formatters or "NOOP"
    insert("if (")
    user.insert_formatted(text, formatters)
    insert(") {}")
    key(left)
    key(enter)

script for overlay commands

from talon.scripting import global_speech_system
from talon import canvas, ui, ctrl, cron
from talon.types import Rect
​
display_canvas = True
​
​
def _draw(canvas):
    print(canvas.x)
    paint = canvas.paint
    paint.textsize = 24
    canvas.clear("ffffff00")
    paint.color = "ffffffff"
    canvas.draw_text(text, canvas.x + 30, canvas.y + 30)
​
​
def reposition_canvas():
    x, y = ctrl.mouse_pos()
    print(x, y)
    can.move(x, y)
​
​
if display_canvas:
    can = canvas.Canvas.from_rect(Rect(500, 850, 600, 50))
    can.register("draw", _draw)
    can.show()
    can.freeze()
​
text = ""
# cron.interval("1s", reposition_canvas)
# reposition_canvas()
​
​
def _log(args):
    global text, can
    text = " ".join(args["text"])
    can.freeze()
​
​
if display_canvas:
    global_speech_system.register("phrase", _log)

script for hissing

from talon import noise, ctrl
def on_hiss(active):
    ctrl.mouse_click()
noise.register("hiss", on_hiss)

script cursor scrolling while hissing

from talon import ctrl, noise, cron
from time import sleep, time
from user.knausj_talon.code.mouse import start_cursor_scrolling, stop_scroll
​
start = 0
running = False
noise_length_threshold = "500ms"
threshold_passed = False
​
def still_running():
    global running
    global threshold_passed
    if running:
        threshold_passed = True
        start_cursor_scrolling()
        print('hiss duration passed threshold, starting gaze scroll')
​
def cursor_scroll_on_hiss(is_active):
  global start
  global running
  global threshold_passed
  if is_active:
    start = time()
    running = True
    cron.after(noise_length_threshold, still_running)
  else:
    running = False
    if threshold_passed:
        threshold_passed = False
        stop_scroll()
        print('end of hiss detected, disabling gaze scroll')
​
​
noise.register('hiss', cursor_scroll_on_hiss)

script simple cron task

from talon import cron
import time
def on_interval():
    now = time.perf_counter()
    print(f"{now:.3f}s")
cron.interval('1ms', on_interval)

script overwrite alphabet

"overwrite" knausj vocabulary with custom cope by adding context

alphabet = "air bat cap drum each fine gust harp sit jury crunch look made near odd pit quench red sun trap urge vest whale plex yank zip"
letters = "abcdefghijklmnopqrstuvwxyz"
from talon import Context
ctx = Context()
ctx.matches = 'os: mac'
ctx.lists['user.letter'] = dict(zip(default_alphabet, letters_string))

script launch sound settings

one can type text by quoting it

sound settings:
    key(super)
    "sound settings"
    key(enter)

script space to pause talon

key(space:down):
    speech.disable()
key(space:up):
    speech.enable()

script up_down_modifiers.talon

hold down <user.modifiers>+:
        key("{modifiers}:down")
release <user.modifiers>+:
        key("{modifiers}:up")

script: move mouse based on screenshot

source: https://github.com/thinium/knausj_talon/blob/bdf222f7/apps/ableton/locate.py

from talon.experimental.locate import locate_hover
from talon import Module, ctrl
mod = Module()
@mod.action_class
class Actions:
    def locate(name: str):
        """Find an image on the screen and put the mouse in the center"""
        locate_hover(name, threshold=0.95)
    def nudge_mouse(x: int, y: int):
        """Move the mouse relatively"""
        _x, _y = ctrl.mouse_pos()
        ctrl.mouse_move(_x + x, _y + y)

script: windows lock screen

source: https://talonvoice.slack.com/archives/C7ENXA7C4/p1629413391497700?thread_ts=1629350122.432300&cid=C7ENXA7C4

from talon import Module
mod = Module()

@mod.action_class
class Actions:
    def sleeplock():
        "Locks the screen"
        from ctypes import cdll

        user32lib = cdll.LoadLibrary(r'C:\Windows\System32\user32.dll')
        user32lib.LockWorkStation()
os: windows
-

lock it up:
    user.sleeplock()
    mimic('sleep all')

vs code plugin cursorless

https://github.com/pokey/cursorless-talon

eye tracking

tobii eye tracker positioning

https://help.tobii.com/hc/en-us/articles/210250305-Position-in-front-of-the-Tobii-Eye-Tracker

[VIDEO] Talon Eye Tracking + Optikey Mouse

https://www.youtube.com/watch?v=PQkJE-rtn-g

taras awesome list

https://gist.github.com/tararoys/480d9b32c137d2f2f882ce06ccf12604

user scripts

alternative/additional software

OptiKey

OptiKey - Full computer control and speech with your eyes

VoiceAttack

https://voiceattack.com/

video showing how to use for programming

mac "bug": dots added randomly while typing

system preferences > keyboard

disable option "add period with double-space"

mac specific: key combinations

on Mac the modifier typically needs to be pressed "with the key" so one cannot

  1. press modifier key :down
  2. press key
  3. release modifier key :up

while on windows/linux this works

e.g. to press n then h while pressing cmd-ctrl: key(cmd-ctrl-n cmd-ctrl-h)

press key for 2 seconds

user.key_async("space", "2000ms")

videos

VIM add-ins/programmes

programmes that enables to control the OS/application in a "vimmy"-way

Firefox

Chrome

Windows

Mac

select text and extend selection

when selecting text one would say select word left

to extend the selection one could say 2nd repetitive to extend the selection +1 word to the left (3rd, 4th, etc would do +2, +3 words, etc)

parrot

quickstart

  1. make ~/.talon/parrot
  2. copy parrot_ensemble_012.pkl to ~/.talon/parrot/model.pkl (from https://github.com/chaosparrot/parrotpy_tryout_bundle)
  3. create a ~/.talon/parrot/patterns.json file
  4. download parrot_integration.py from #beta and move it to ~/.talon/user/
  5. create a .talon file based on the pdf
  6. make the sounds

v2 changes

  1. updated parrot_integration.py
  2. update patterns.json: s/f1/f0/

A parrot testing file

source: https://talonvoice.slack.com/archives/C01QEDFH60J/p1627679500015900

parrot(cluck):
        "cluck"
parrot(tut):
        "tut"
parrot(palate_click):
        "palate_click"
parrot(pop):
        "pop"
parrot(gluck):
        "gluck"
parrot(finger_snap):
        "finger_snap"
parrot(ch):
        "ch"
parrot(ah):
        "ah"
parrot(oh):
        "oh"
parrot(ee):
        "ee"
parrot(ue):
        "ue"
parrot(ae):
        "ae"
parrot(horse):
        "horse"
parrot(whistle):
        "whistle"
parrot(chopper):
        "chopper"
parrot(hurr):
        "hurr"
parrot(hmm):
        "hmm"
parrot(oo):
        "oo"
parrot(lll):
        "lll"
parrot(yee):
        "yee"
parrot(uh):
        "uh"
parrot(fff):
        "fff"
parrot(buzz)
        "buzz"
parrot(generator):
        "generator"
parrot(hiss):
        "hiss"
parrot(shush):
        "shush"
parrot(x):
        "x"

action after noise ends

parrot(hiss:stop): print("hiss stop")

action when noise starts

parrot(hiss:start): print("hiss start")

start/stop noise recognition

actions.mode.disable('noise')
actions.mode.enable('noise')

tryout bundle

https://github.com/chaosparrot/parrotpy_tryout_bundle

debugging

test command [REPL]

using the sim() function

Talon REPL | Python 3.9.4 (default, Apr 24 2021, 16:58:24) [GCC 7.5.0] on linux)
>>> sim("slack close right sidebar")
[1] "slack close right sidebar"
   path: user/gpunkt_talon/mode-all.linux.talon
   rule: "slack close right sidebar"

show what events where triggered [REPL]

events.tail()

search talon repos

https://search.talonvoice.com/

conformer

version history

  • a74 - first/original release
  • a108 - second release
  • b108 - third release: changed packaging but not performance

in Talon >= 0.2.0-8 conformer can be used without a language model (by renaming lm-ngram.bin)

alternative movement command

instead of go up fifth one can say go up five times

options

with b108 there is a torch option

engine = W2lEngine(model='en_US-conformer', debug=True, torch=True)


mac key "option"

use 'alt' instead of 'opt' or 'option' - example: key(shift-option-down)

set command timeout

settings():
    speech.timeout = 0.3

german.talon - switch between talon (en) and dragon (de)

source: https://github.com/wingerath/talon_german/blob/wolle_personal/german.talon

mode: user.german
-


^(englisch | english)$:
    user.engine_mimic("geh schlafen")
    mode.disable("user.german")
    user.enable_withSound()

^(snore)$:
    user.engine_mimic("geh schlafen")
    user.sound_disable()
    user.app_color_black()


talon sleep | snore | go to sleep: skip()

^german$: user.sound_disable()

Umbruch: actions.key("shift-enter")

engine.py

from talon import speech_system, Context, actions
from talon.engines.w2l import W2lEngine
from talon.engines.vosk import VoskEngine
#w2l = W2lEngine(model='en_US', debug=False)
w2l = W2lEngine(model='en_US-conformer', debug=False)
#vosk_de = VoskEngine(model='vosk-model-de-0.6', language='de_DE')
speech_system.add_engine(w2l)
#speech_system.add_engine(vosk_de)
# # ideally you integrate this into your engines.py file
#1
# from talon import speech_system, Context
#
#
# # especially this should not be here but in your engines.py file:
ctx = Context()
ctx.settings = {
    'speech.engine': 'wav2letter', # your default engine goes here
#    'speech.engine': 'dragon', # your default engine goes here
}

control zoom mouse scale

https://search.talonvoice.com/search/?q=zoom_mouse.config&fold_case=auto&regex=false&context=true

{number_small}

one can use numbers like key("cmd-alt-{number}") but if that number won't exceed 99 one should use {number_small} which caps at 99 whereas {number} is capable of infinite numbers (e.g. one hundred trillion)

mixed mode

source: https://talonvoice.slack.com/archives/G9YTMSZ2T/p1641064850309500

dictation_mode.talon, add this at the top:

experiment: anchor-file

modes.talon, add at the bottom:

^mixed mode$:
  mode.disable("sleep")
  mode.enable("dictation")
  mode.enable("command")