Python-Editline¶
A Python extension which links to libedit.so and implements the shell-completion functionality completely independent of libreadline.so and readline.
My operational goal was originally to try to get the readline functionality of libedit to “just work”. Apparently me, and a bazillion others have tried and only got something marginally functional.
I changed the plan. For reasons I do not know, the ‘readline’ interface of Python is heavily favouring the libreadline.so interface. In my view, that is a problem on several fronts. It makes portability to libedit a pain, and have you seen how many freeking global variable libreadline uses just to function? It is crazy.
My (so far successful) approach has been to abandon the readline api and work towards a generic line-editor interface to which both libedit (and readline if necessary) will conform.
The goals are:
- to create a Python installation devoid of GPL
- run correctly on Linux, *BSD, SunOS and Mac
- not muck up the current terminal
- use libedit either built-in on various BSDs or use http://thrysoee.dk/editline/
- push it back into Python main-line
- Move the bulk of the “completion” functionality back into Python code where it is a hell of a lot easier to manage lists of strings
Current State¶
I have done various tries at this. The module/ directory is so far the one that works.
Basic tab completion runs smoothly (… in Python code)
Enhanced features of tab-completion
- Across lists (index completion and view)
- Across dicts (key-completion and view)
- Support for tab-completion in any complexity of command
interface starts up automatically with ‘python -i’
system-wide editline instance is created automatically
terminal resizing works
support for RIGHT-SIDE prompt (as well as std) is present (I don’t think readline can even do that)
support for ^c and ^d match the readline functionality
no terminal changes occur after running
history support works
- adding commands
- loading history file
- saving history file
setup.py will construct either an extension which refers to libedit.so or will build a larger extension with the libedit sources embedded.
support for “history” and bash-like shortcuts to access historic cmds
significant test cases have been implemented
Installation¶
Simple Case – No sitecustomize.py in python or virtual-env¶
To get this thing working, I recommend this:
# cd /handy/location
# git clone https://github.com/mark-nicholson/python-editline.git
# make
# python3 -mvenv myvenv
# . myvenv/bin/activate
(myvenv) # python3 setup.py install
(myvenv) # python
>>>
The ‘gnu make’ is there to download and prep the libedit distribution.
More Complicated Situation¶
The installer has support to be able to work-around and merge (sort of) the pre-existing sitecustomize.py file and the new necessary changes.
# cd /handy/location
# git clone https://github.com/mark-nicholson/python-editline.git
# make
# python3 -mvenv myvenv
# . myvenv/bin/activate
(myvenv) # python3 setup.py build_py --merge-sitecustomize
(myvenv) # python3 setup.py install
(myvenv) # python3
>>>
The second situation will create a backup of the original sitecustomize.py insitu.
A note of caution before you pull all of your hair out. SOME distributions (pointing at you Ubuntu) drop in a sitecustomize.py file in /usr/lib/python… which is EXTREMELY annoying as it completely prevents a VirtualENV from creating its own ‘customization’ using this because the system path is always ahead of the venv path. sigh. You can hack this by renaming it to usercustomize.py and dropping it at the same place – ugly but effective.
I tested this in both a default install and virtual-env. The instructions detail the VENV method which I used most. The sitecustomize.py file is critical as it is the init-hook which disengages readline as the default completer and installs editline.
Another way to make it work in a default install, you can edit site.py and replace the enablerlcompleter() function with the enable_line_completer() function from sitecustomize.py in the repo. It does not matter whether you do it in the cpython source tree, or after it is installed. It is coded in such a way that if editline support is absent, it will still fall back to readline.
Situation - I’ve got a bork’d libedit, how do I force the builtin?¶
For this case, the default install will hopefully figure out that
libedit is bork’d and all is well. If it doesn’t, then you’ll have
to manually force it.
# cd /handy/location
# git clone https://github.com/mark-nicholson/python-editline.git
# make
# python3 -mvenv myvenv
# . myvenv/bin/activate
(myvenv) # python3 setup.py build --builtin-libedit
(myvenv) # python3 setup.py install
(myvenv) # python3
>>>
The --builtin-libedit option to the build phase overrides the
system snooping. It will still do some checks to ensure it has enough
stuff available to build the builtin-libedit (and link it).
How do I check?¶
>>> import _editline
>>> gi = _editline.get_global_instance()
>>> gi.rprompt = '<RP'
>>> <RP
>>>
Get readline to do THAT! (ok, if your browser is narrow, you should have the ‘<RP’ on the other end of the command line – the interpreter should do it right)
Premise? Ok, so the editline infra uses instances, NOT GLOBALS. The get_global_instance() API gives you “roughly” the same thing as ‘import readline’. We will see if we can smooth it out a bit more later.
Annoyances¶
I am vexed and cannot understand why Modules/main.c in the Python build does this
v = PyImport_ImportModule("readline");
I’ve commented it out and found no alteration of functionality. It probably is some legacy code which was put in in the dark-ages, and just doesn’t piss anyone off…. except me. Its presences seems benign, but it still concerns me. Even when, disconcertingly, both readline.so is loaded (by main.c) and _editline.so is loaded by the standard method, this then has both libreadline.so and libedit.so loaded into the process. There will certainly be a symbol-collision for ‘readline()’ (in C) as both have it. I suspect that it will work without any issues since editline.so only calls the libedit API (el*) and does not reference the C call to readline() in libedit. Tread carefully.
If you are rolling your own Python interpreter, patch out the silly import in main.c. Then, even if you have both readline.so and _editline.so, they will be mutually exclusive! (Check the patches/ directory.)
Tasks¶
The baseline code is working. It is considerably simpler than the readline implementation and also uses instances (even the ‘system’ one) so it can support multiple instances per interpreter.
- Create changeset for Python codebase (integrate everything there?)
- Implement more test infrastructure to beat it like a rented mule
- build out functionality for
- bind()
- el_get()
- el_set()
- history()
- create a mapping routine so ‘parse-and-bind’ can support a common command format and translate to the “local” flavour.
- potentially create a subclass which is a readline drop-in-replacement.
Acknowledgements¶
Thank you to:
- http://thrysoee.dk/editline/ for creating the portable version of the NetBSD ‘libedit’ library.
- the libedit developers who made the examples/ directory. Great baseline code.