This round of work on Jambu concentrated on developing 'direct in application' control from switches and other limited gesture Alternative Input devices. The idea is to provide a mechanism to operate a program's User Interface directly with without having an intermediate overlay (or OSK). This direct navigation and selection of applications provides access to all features and should therefore be attractive to users who are confident computer operators but have (or have developed) restricted movement. Mozilla have again provided funding through their accessibility grants scheme and I'm very grateful to them for making it possible. This is both a work report for Mozilla and a personal commentary of my experience developing an AT.
Eitan Isaacson has acted as a mentor and brought his invaluable experience with pyatspi from his work on accerciser and Orca. He kindly provided some example code (hacked on a domestic flight) that demonstrated a technique for minimal caching, suggested using GTK TreeView to give a starting MVC architecture and also help 'knock up' a proof-of concept of the code to draw over an application. I'm very grateful for his input during a time when he was very busy working on improving Orca's performance. In addition members of the wider Open Source accessibility community provided great help and ideas (this is one of the really positive features of Open Source and one I should tap into more).
So what can it do? It provides access to most of Minefield's UI and so proves the concept as well as providing a firm bases for further development into a polished solution. There are a couple of bugs to fix (they always appear after you've made a release). Some improvements to make and a few temporary work-arounds waiting for Mozilla bugs to be fixed or where work is in progress on a feature. One of the practices of Open Source development that I find mildly disconcerting is releasing early, warts 'n' all, so others can (potentially) test and review it. Being a bit of a perfectionist I naturally want a highly complete solution before I release it on an unsuspecting world. You do have the same tension in proprietary development too, as at some point you have to get it out the door, but my experience is that you usually wait as long as you can, largely as the user is paying for it and releases are less frequent.
The most important lesson I learnt is that creating an AT is a challenging task with many complexities and subtleties to deal with ( "Nuff R'spect" to the Orca and GOK devs). Accessibility APIs are rich and application structures and behaviours are complex systems with a few 'gotchas' to be worked out. I endeavoured hard to avoid lots of case specific logic and keep to generalisations that minimized dependencies but that is hard to achieve. In hindsight I could have just hacked it all out and then worked out the emergent rules (the would still have hit the same problems, but saved time trying to get good code). A certain amount of 'scripting' is going to be inevitable to handle application differences (I plan to use a template design pattern).
At times it gets frustrating as you yet again revisit code that you thought was working but breaks when implementing a new feature or fixing a bug. This reminds me of the old Windows development saying of how 'when you empty the ashtray the wheels fall off'. I don't mean to sound negative as this is all part of the learning curve for developing accessibility and you develop techniques to avoid it. If it was easy then everyone would be doing it already (there's plenty of technical fun to keep us fully engaged). The curve proved much steeper than I expected and so took longer than investigative test scripts indicated. The trick will be to help new AT developers to short circuit the learning curve.
I started out on the wrong foot to some extent, suffering endless desktop lockups and apparently non-deterministic behaviour. The code was initially created as an accerciser plugin as that gave a useful environment and easy ways to explore with the possibility of links to accerciser functionality. I was also using the excellent Wing IDE debugger to explore and step through the code (and I love the DebugProbe feature that lets you modify and prod the live system). At Eitan's suggestion I started with GEdit as the target because GAIL is pretty stable and that would would provide better visibility on what issues were infact Firefox bugs.
It turns out that AT-SPI events are synchronous (direct procedure calls, not asynch RPC as I sort of assumed). So you have to do as little as possible in the event handlers and return ASAP. (this is rather like the good old days of Interrupt Service Procedures or Real Time, without the reentrancy and latency issues). GAIL is also a little sensitive to what you call when and can easily go into a sulk. Anyway once I made a stand alone application, desynchronised with a queuing mechanism, stopped using the debugger and reverted to good old print statements things got a lot more stable. There are still timing issue when printing a lot of data and other timing funnies but these will get resolved with attention to the structure and state processing.
Popup menus and windows are the most complex and sensitive to work with and of course I started with them. Most user actions make them disappear (by design) so they are hard to work with and monitor in accerciser. Little exploratory test scripts help. The massive effort on Orca (and other screen readers) means that the keyboard driven, focus monitoring dynamics are well understood and debugged. However Jambu works by calling the doAction function without focus setting the context and this uncovers a few strange behaviours (aka bugs) which had to be understood, bugs raised and workarounds found. Sometimes events just don't seem to happen or happen in different orders so you need to code around that. You also don't see all of the events that you might expect. Working on the trunk also exposes you to changing features which can be confusing at times (it's a trade off to wanting bug fixes and wanting to be isolated from other changes).
Accerciser, GOK and Orca code provides examples and inspiration (though I initially missed the significance of the event queues). You can also look at the Gail and Firefox code to try to understand what is happening but it is complex anda little obtuse (Ieven attempted to fixed a Firefox menu bug but with no result yet).
Future work includes cleaning up the code and structure, possibly introducing a formal state machine to handle the dynamic behaviour better and definitely adding per toolkit/application scripting. A better story is need for web documents, in particular dealing with scrolling large documents and handling large sets of links/fields. Multicolumn lists (table/tree widget) needs some TLC in order to have sensible navigation and for selection to work. Text input needs integration with Dasher/GOK or OnBoard (someone has said they are interested in looking into Dasher). To be a usable AT more scanning options are needed as well as clever navigation schemes and smarts as it is clear that the current simple scheme can be long-winded at times.
In response to Eitan's suggestion of regular releases I spent a little time setting up a source release (using distutils and providing NEWS, ChangeLog etc), bringing this forward as I had it planned as a future task. There's also a jambu google group to aid communication. This has the benefit of making it more accessible project, with clearer progress and so hopefully attracting others who can bring their ideas, experience and help (hint, hint).
